Driven by the increasing diffusion of embedded sensors and actuators, and more in general by "Internet of Things" (IoT) devices, pervasive computing is becoming a reality. Yet, most actual implementations of pervasive computing environments rely on rather centralized architectures and on middleware solutions that integrate only the minimal set of services to enable interoperabilty and data integration. In this article, after having overviewed the state of the art in the area of pervasive computing middleware, we discuss the many challenges that still have to be faced for pervasive computing middleware to be able to support elastic, easy to configure, easy to develop, safe, and ethically acceptable, pervasive computing services and applications.