Including code from non-free libraries will make your package non-free.
Not true. The result would be illegal, not non-free. For it to be legal the whole work has to be licensed under the GNU GPL.
Once can add a special execption to the GPLed library so one allows for non-free libraries to use it, see the GPL FAQ "How can I allow linking of proprietary modules with my GPL-covered library under a controlled interface only?"
The GPL FAQ has more answer releated to this.