The last two episodes of series 4 do pick up, but I also felt the same up to this point.
I enjoyed the series overall but not as much as the previous ones. But in years to come when the whole show is complete, that's not going to matter. 4 will just be the series people say is a bit slow, but it's necessary and still enjoyable.
Like season 2 or possibly 3 of Breaking Bad is pretty slow; so much so that people stopped watching it completely, but the 5 seasons of BB are considered up there with the best TV ever.