If by "design side" you mean civil/structural/geotechnical engineering, it can work out quite well if your credentials allow you to progress to professional registration & a PE license. If not, it would be best to move into the management side of design as much as you can, where the work is more about coordinating w/ contractors, rather than technical details. There is currently a large portion of "design side" engineering being outsourced to BIM modelers, who sometimes (most of the time) have very little experience in actual construction or project management. Project deliverables would experience a huge drop in quality if competent personnel weren't overseeing the technical details and coordinating documentation.
Experienced construction industry people tend to know about site prep, material handling/traceability, and selecting the best/realistic resolutions to disputes. Those are important traits that avoid damaging the client's wallet. You'll need to find a mentor who can help guide you at the beginning of a new path, so you don't spin your wheels and lose earning potential.
TLDR: Yea. If I can do it, you can too. Use time judiciously.
Edit to add my path:
Started w/ construction-side project engineering and scheduling. This was one of those jobs which was "engineering" in title only. It was mostly procurement, tracking materials in delivery and laydown, monitoring concrete pours, ensuring QC reps got everything they needed, etc. After a few years, I went to a steel fabrication company as a design engineer. Passed the PE exam and got credentialed and good money bumps. Not a ton of advancement available in that particular position. Currently making plans to move towards a project management role where I can combine the knowledge bases at the lead/senior level. You can do it.