Solar panels are indeed a worthwhile investment for off-grid living, however, they aren't without their flaws. Like BlueMarigold pointed out, they aren't as efficient during overcast or winter months. To combat this, consider investing in a good quality backup power generator or a second type of renewable energy system like a wind turbine.
Apart from this, another downside is the initial cost. Solar panels and the required setup (including batteries, wiring, switches and so forth) can be quite expensive. Don't forget, you may also need to replace the