Mostly the satellites transmit at one of two bands, at 144 MHz or at 433 MHz. I had dipole antennas built for the bands already, but I didn't have a balanced feed to them as I had been using them inside very close to the receiver. To have the best chance of receiving anything from the satellites, I had to place them outside. The best solution of course would be to have the receiver reside outside close to the antenna and to just pull the data through a cable, but high speed USB doesn't really work over very long cables. The other solution would then be to have a computer (like a raspberry pi or something) close by to the receiver, perhaps even outside, and then pull the data in over Ethernet. The distance would have been too long for USB and I don't currently have a suitable small computer to use near the receiver I couldn't do with either of those options. Thus I had to add some balancing to the antenna feed and to pull the signal in with a coaxial cable.
The simplest way to build a balanced antenna, I found, was to build a folded dipole with a loop of coaxial cable acting as the balun. The folded dipole has an impedance of about 300 ohms, with the coaxial cable doing a 1:2 voltage division, thus acting as a 1:4 balun. The feed signal is then a 75 ohm signal, which is suitable for the RTL-SDR. A folded dipole is also basically as easy to build as a regular dipole.
With this new setup I was able to hear the local transmission at much increased power and signal-to-noise ratios. However, I didn't hear any satellites. After several nights of trying to hear something, I realized that the amateur radio satellites were in fact solar powered, and not transmitting when not illuminated by the sun. Unfortunately, I don't get much chance listening to satellites at day time due to my day job, and even during weekends it is only possible when the kids are napping.
One evening, just a short while before sunset, I plugged in the receiver to notice a strong transmission I hadn't seen before. Looking at it more closely, I realized it was drifting in frequency! This had to be a satellite. Unless someone was deliberately generating a signal that looked like it had such a strong Doppler shift. I started recording the IQ samples to store it for demodulating later. After two minutes of recording, the signal had vanished.
As the signal didn't want to be demodulated, and as I couldn't even figure out what modulation it uses, I started to investigate which satellite was responsible for the transmission. The original transmission frequency is difficult to estimate due to the Doppler shift, but my guess is that it was somewhere in the band between 435.56 MHz and 435.57 MHz. I first looked at the amateur radio satellites in GPredict, as it has a very nice time control feature, which easily allows me to check what satellites were in the sky at the time of the transmission. Turns out that GPredict doesn't know this satellite, as none of the satellites it showed transmit at this frequency.
Turning to Google and looking through lists of satellites and their transmit frequencies gives a short list of possible candidates. Finally I find a match with a Russian military satellite Kosmos-2499! It carries a radio amateur payload called RS-47, which transmits on 435.565 MHz, and which was on the sky at the time of recording. There is also this YouTube video with the demodulated signal from the satellite. It sounds exactly like the signal I got. That's pretty cool.
The international space station transmits constantly, even when it is in the Earth's shadow. So in the evening I tuned the receiver to the ISS's frequency range and started to record a spectral histogram. In the morning I could clearly see signals corresponding with ISS passes, as well as several other transmission which also were clearly from satellites.