BlackBerry QNX’s self-driving Lincoln MKZ – what’s under the hood?

Kerry Johnson

Sr. Product Manager

BlackBerry QNX

At CES 2017, BlackBerry-QNX unveiled its self-driving Lincoln MKZ. 

In years past , BlackBerry QNX has become known for displaying its innovative technology in its concept cars, which included infotainment, mobile device connectivity, digital instrument clusters and ADAS. This year BlackBerry QNX has outfitted a Lincoln MKZ to demonstrate a self-driving vehicle. The Lincoln MKZ is much more than a demonstration vehicle – it is an engineering prototype that allows BlackBerry QNX engineers to experiment with and develop new technologies for the autonomous vehicle market.

You may wonder why BlackBerry QNX chose a Lincoln MKZ for its autonomous driving car. The reason is straight forward. The 2017 Lincoln MKZ comes equipped, from the factory, with all the necessary drive-by-wire capabilities. All of the driving systems (throttle, gearbox, steering and braking) can be completely controlled electronically. By using this capability as a starting point, BlackBerry QNX and its partners are able to focus on adding other self-driving capabilities such as the sensors, route planning, and maneuvering.

While providing the foundational software, BlackBerry QNX did not build this self driving vehicle alone. We worked closely with Renesas, University of Waterloo, Polysync and Cogent, to put the car on the road.

The following is a brief walkthrough of the technologies inside the Lincoln MKZ:

BlackBerry QNX

BlackBerry QNX’s goal was to build an autonomous vehicle using commercial embedded processors and safety certified embedded operating system (OS). At the core of the design was QNX’s safety certified OS, which powers all of  the intelligent software modules. QNX’s middleware serves to integrate RADAR, LIDAR sensors , multiple camera inputs and vehicle networking. BlackBerry QNX provided a port of the OpenCV library to help with the vision processing functions delivered by Cogent. 

BlackBerry QNX also provided a port of Robot OS (ROS), so that the University of Waterloo could easily bring their self-driving software algorithms to the car without having to re-write large portions of code.

The ROS software components are not truly embedded, production oriented software. However, in building an autonomous car we chose a phased approach. We chose to use existing software to test and validate the solution. This saves time and allows flexible prototyping. Once the code is finalized we can convert it into an embedded solution.   

University of Waterloo

The University of Waterloo, one of Canada’s leading autonomous driving research institutions, contributed several software components, including static and dynamic environment perception, path planning, maneuvering and dispatching control commands to the various actuators. It should be noted that, at the outset of the project, the University of Waterloo already had a number of these components operational. Part of the activity was to port the software from Linux to QNX – a task made simple by BlackBerry QNX’s support for the POSIX standard.

Polysync

Polysync provided their framework for distributed communications and sensor integration. They also provided system data visualization tools, so the engineers could see how the system was operating from a central console.

Cogent

Cogent provided a number of vision processing algorithms that processed input from multiple camera sensors.

Renesas

The compute horsepower in the Lincoln MKZ comes from two Renesas R-Drive reference boards. Each Renesas R-Drive board has two Renesas R-Car system on chips (SoCs), each with quad-core ARM processors and image processing accelerators. Two R-Drive systems were used so that fail-over scenarios could be tested.

Sensors

The following sensors were used to construct a 360-degree view of the surroundings and to achieve accurate positioning of the car:

• 1 Delphi long range radar
• 1 Delphi short & medium range radar
• 2 Velodyne LIDARs
• 1 forward-facing Point Grey camera
• High precision GPS and IMU (Inertial Management Unit)

The car is now running on a test track.  In the following years BlackBerry QNX will continue to refine the system towards production oriented hardware and software. 

Read More

From Concept to Reality: BlackBerry-QNX's Groundbreaking CES Tradition

Thomas Bloor
Business Development Manager, QNX BlackBerry

The annual Consumer Electronics Show in Las Vegas has been growing in importance for the automotive industry over the years. You can hardly fail to notice that this year, as in previous years, the big automakers vie for floor space and attention with the glut of big screen TVs and other consumer goods. As always, BlacBerry QNX will be in the North Hall, proudly in the middle of the big automotive OEMs. 

At CES BlackBerryQNX has an enviable history of bringing concept cars that rival anything on the show floor, with one important difference – ours are not pure flights of fancy, and we show technologies that will become realities in the near future.

We started this trend back in 2010 with an LTE-connected Toyota Prius – 18 months before the first commercial LTE deployment in mid-2011. Working with Alcatel-Lucent to provide the experimental network, we demonstrated Google maps functionality with local search and an embedded Pandora radio app in a car for the first time. Connectivity is standard in many cars today, but in 2010 we demonstrated the future.

2012 brought us a CNET "Best of CES" award for demonstrating cloud-based natural language voice recognition, text-to-speech, and NFC based one-touch Bluetooth pairing.  Simply touching your phone to an NFC reader in the center console automatically paired the phone and car. 

In 2013 we got ahead of the trend for ever larger center stack displays – with detailed 3D maps and voice recognition Keyword Spotting – common today in smartphones but a first in a car. Simply saying "Hello Bentley" enabled you to start interacting with the natural language cloud based voice recognition Powered by AT&T’s Watson. 

2014 took literally us in a different direction. A 21-inch horizontally orientated center stack display extends across the dash, naturally extending the interaction and functionality towards the passenger.  Behind the screens the instrument cluster was integrated with the center stack running both driver information and IVI functions. With seamless controllability across the touch screen, physical buttons, and the jog wheel controls multi-modal input was highlighted across all available functionality. 

Not content with that, we foreshadowed greater integration of ADAS functionality warnings to the driver. In 2014 we warned the driver if local speed limits were exceeded through both the cluster and verbally through text-to-speech, and we followed this up in 2015 with a system that recommends an appropriate speed for upcoming curves based upon driving conditions and the radius of the bend.

So, what innovations will we be showing in 2017? I’m not allowed to tell you just yet but, in a first (for us), we’ll be showing both future and current production technologies and innovations.

Building on our products ranging from in-car acoustics through our comprehensive QNX-CAR application platform and to next generation driver assistance/autonomous drive we will be demonstrating how technology can enhance the user experience and increase safety for drivers and passengers.

While demonstrating technologies that will come to future production vehicles, these cars are not just "show floor wonders" because our automotive knowledge enables us to build demonstrators for the real world, which can be driven, thus allowing technologies to be experienced first-hand.

Read More

Holistic Security for the Software-Defined Car

Bill Boldt
Sr. Business Development Manager, Security
Blackberry Certicom

Due to high profile hacks on cars, it is hard to argue that without security you can have safety.   So, security is emerging as perhaps the most important factor in the evolution of the connected autonomous car.
 
Cars are the most software intensive systems in the universe with far more lines of code than even a state of the art jet fighter. By being such complex digital systems they have become prime targets for attack, and that is where cryptographic countermeasures come in.

Connecting the dots – in the emerging software-defined world safety increasingly
comes from security and security comes from cryptography. Robust cryptographic security implementation is how you increase trust, and when it comes to a car every system must be
trusted: inside the car, in the smart infrastructure, in emerging applications-based ecosystems, and in the manufacturing supply chain. When considering automotive security,
many factors come into play. Some are noted here:
                       

• Automotive security fundamentally depends on the security of the operating system. For example, a microkernel architecture that separates critical OS components into their own protected memory partitions, provides temporal separation, and provides network security, among other things can greatly reduce the attack surface.

• Security assets (crypto keys, serial numbers, etc.) must be securely installed into electronic devices such as Electronic Control Units (ECUs), domain/area controllers, and other processors. This process is called "personalization".

• Electronic devices will often get personalized and installed into vehicles in globally located factories, which should utilize secure equipment and processes to ensure security of the devices.

•  Devices must be updateable at dealers and repair shops. 

• Aftermarket suppliers must be able to sell and update secure devices, and

• OEMs must be able to authorize or not authorize specific electronic devices at
  manufacturing time and after the car is in use (for example to enforce warrantee policies).
  

And, there are many more.


Personalizing a device such as a networked ECU means that it will become one of a kind. However, by definition that device cannot be used anywhere else. It becomes a unique stock keeping unit (SKU), which is averse to the purpose of flexible, just in time manufacturing flows. Security versus manufacturing flexibility is a serious trade off that will play a part of any automotive security design decision.

Security robustness versus cost is another critical trade off, and applies to the manufacturing infrastructure and the design of the secure systems inside and outside the vehicle. Because security must be injected in the factory and in the field, a secure manufacturing system must have global reach, be manageable on a distributed basis, be updatable by various entities, and remain secure for years. In addition, security updates will increasingly be made over the air, and the systems that do that must by highly secure while being easy to manage. To maintain the maximum amount of flexibility, personalization and updating should be moved as close as possible to the very last minute, which is becoming a critical objective of the global manufacturing blue print. 

Blackberry Brings It All Together

In the car, outside the car, and in the manufacturing supply chain, security must be designed with best practices in mind right from the start, and BlackBerry Professional Services can help with that. BlackBerry QNX provides mission-critical automotive software proven in the automotive market.  QNX software is well known for safety and new products are setting the new standard for security.

BlackBerry's Certicom subsidiary provides certified cryptographic code and design consulting, as well as secure equipment and managed services that harden the automotive supply chain. Completing the picture, BlackBerry's secure OTA managed services make it easy to update software and security assets over the air. When it comes to automotive security, BlackBerry brings it all together.

Read More

QNX's Fabulous Concept Cars

Thomas Bloor

Business Development Manager, BlackBerry

If you’ve been to CES chances are you’ve seen one of our concept cars, even if you’ve not been to our booth, we take these to our industry partners. Now as we start gearing up for next year’s CES there are some great innovations in the pipeline. (I’ve seen them, but I’m not telling.) So as I can’t spill the beans on what's coming, but let’s take a look at some of my favorite QNX Concept Cars from years past.

The Porsche 911 Carrera (CES 2012)

Admittedly I have a soft spot for performance cars, but the Porsche deserves headline billing in the roster as a CNet "Best of CES" winner. With revolutionary (for 2012), cloud-based voice recognition you could control the navigation system using natural language.   And, text-to-speech meant that you could listen to incoming BBMs, emails and text messages. Rounding out the roster of features that would still be considered ahead of the curve for a production car today, this model featured one-touch Bluetooth pairing.  Simply touching your phone to an NFC reader in the center console automatically paired the phone and car. 

The Bentley Continental GT (CES 2013)

In an outburst of Canadian quirkiness, we decided that when better to do a photoshoot of a Bentley Convertible than in the middle of the Canadian winter? Of course despite the -20C (-4F) weather we’d have to have the top down!

The cold and the snow do not detract from the revolutionary center stack with DLP® display from Texas Instruments. This immense (for 2013) featured an organically curved surface and TI’s optical touch input technology, which allowed physical control knobs to be mounted directly on the screen resulting in an ideal balance in physical and touchscreen controls

Taking natural language voice recognition a step further we worked with AT&T’s Watson^SM . Say "Hello Bentley," and the car's voice recognition system immediately starts interacting with you, in a distinctly British accent, old chap.

If that weren’t enough, the cluster displays the back-up camera and user configurable high resolution instrumentation. We also took the mobile office to new heights with smartphone integration with streaming music, email notification, news feeds, and other real-time information. Put the Bentley into park and you could fire up video conferencing with realistic telepresence.  

Separate cameras for the driver and passenger provide independent video streams, while high-definition voice technology from QNX offers expanded bandwidth for greater realism, while stereo telepresence makes the remote caller sound as if they’re sitting right next to you.

Mercedes CLA 45 AMG (CES 2014)

Have you looked inside a Mercedes S class recently? The horizontally orientated center stack display extends across the dash. Coincidentally our 2014 Mercedes concept had a 21-inch-wide center display extending towards the passenger enabling a seamless interaction with the vehicle.

 

Behind the scenes the Cluster was integrated with the center stack running both driver information and IVI functions. With seamless controllability across the touch screen, physical buttons and the jog wheel controls multi-modal input was highlighted across all available functionality. 

Not content with that, we foreshadowed greater integration of ADAS functionality warnings to the driver through both the cluster and verbally through text to speech if the local speed limits were exceeded.

Jeep Wrangler and Toyota Highlander

Now it’s not all high end luxury cars, which is just as well because they never let me drive any of them. Our Jeep Wrangler and Toyota Highlanders serve as our QNX reference vehicles showcasing what the QNX CAR application platform can do, straight out of the box. Additionally, the Toyota features our advances in in-car communication and acoustics platforms enabling an enhanced user experience for drivers and passengers.

These cars are not just  "show floor wonders" because our automotive knowledge enables us to build demonstrators for the real world, which can be driven, and the technology can be experienced first-hand. Concept clusters and displays abound, but real vehicle bus integration means these cars are drivable with real instrumentation and connectivity.

While I can’t reveal what new exciting technologies we are planning for CES 2017 (believe me, you’ll want to come and take a look), I can say that our reference vehicles are currently on tour so keep an eye open for them on the roads near you.

Read More

“I don’t know where I’m going from here, but I promise it won’t be boring”

Patryk Fournier

The quote is from the now late but great David Bowie and is extremely prophetic when you apply it to autonomous driving. Autonomous driving is very much still uncharted territory. Investments in roadway infrastructures are being made, consumer acceptance is trending positive, and, judging by the news and excitement from CES 2016, the future if anything will not be boring.

CES 2016 stretched into the weekend this year and ICYMI there was a lot of compelling media coverage of QNX and BlackBerry. Here’s a roundup of the most interesting coverage from the weekend:

ARS Technica: QNX demos new acoustic and ADAS technologies
The crew from ARSTechnica filmed a terrific demonstration of the QNX Acoustics Management Platform and the QNX Platform for ADAS. The demonstration highlights the power and versatility of the acoustics platform, including the QNX In-Car Communication module, which allows the driver to effortlessly speak to passengers in the back of the vehicle, over the roar of an engine revving at high speed. The demonstration also showcases how the QNX OS can support augmented reality and heads-up displays:

Huffington Post: CES 2016 Proves The Future Of Driverless Cars Is Promising
Huffington Post highlighted BlackBerry and QNX as key newsmakers for advancements in driverless cars. The article notes QNX’s automotive leadership: “The software is actually installed in 50 per cent of the world’s automotive infotainment systems including Audi, Volkswagen, Ford, GM and Chrysler.”

Crackberry: Inside the QNX Toyota Highlander at CES 2016
The folks at CrackBerry filmed a demonstration of our latest technology concept vehicle, based on a Toyota Highlander. The demo focuses on the QNX In-Car Communication acoustics module, which forms part of the recently launched QNX Acoustics Management Platform:



HERE 360: QNX and HERE bring to life a multi-screen experience in vehicles
A blog post from our ecosystem partner mentions HERE navigation and its use in the Toyota Highlander and Jeep Wrangler technology concept vehicles.

Read More

Why is software the key to bringing augmented reality to cars?

Guest post by Alex Leonov, marketing director, Luxoft Automotive.

While self-driving vehicles are gradually becoming a reality, more and more of today’s cars roll out from factories featuring advanced driver assistance systems (ADAS). We are quickly getting used to adaptive cruise control, blind spot monitoring, parking assistance, lane departure warning, and many other features that make driving safer and the driver’s job easier. Data from cameras, sensors, and V2X infrastructure feed into ADAS systems, increasing their accuracy and efficiency. These systems are important steps toward fully autonomous driving, but the ultimate responsibility for decision making still lies with a driver.

The more that cars become connected, the more the average driver can be bombarded by information while driving. “In 500 feet make a right turn.” “You have an incoming call from Christine.” “You have a new message on Facebook.” “You are over the speed limit.” This may not be so big of a distraction under normal conditions. But sometimes, when driving in hectic city traffic or in a snow storm, it is critical to keep eyes on the road, while still receiving essential information. The good news is, the technology is already there to remedy this.

Heads up for HUDs
Keeping the driver’s eyes on the road is a priority, and head-up displays (HUDs) can accomplish just that. They project alerts and navigation prompts right on the windshield. Analysts predict an explosive growth of HUDs with the market reaching close to US$100 billion by 2020. The bulk of HUDs are relatively simple combiners, but more advances in wide-field-of-view HUDs are coming soon.

Projecting alerts and navigation prompts directly on the windshield.

HUDs are perfect for presenting information in a convenient, natural way, and giving the driver a feeling of being in control. But HUDs are only as good as the information they display. That is why it is critical to have solid and reliable data processing and decision-making algorithms, running on a reliable OS, that can prioritize and filter data. The resulting alerts and prompts must be communicated to a driver in a clear, transparent way.

Computer vision, also known as machine vision, is a key to processing the endless flow of data. With its human-like image recognition ability, computer vision processes road scenes, and the system fuses data from multiple sources. Add in a natural representation of processing outcomes in the form of augmented reality, while tracking driver’s pupils, and you have a completely new level of driver’s experience — safe and intuitive.

Next-generation driving experience
At Luxoft, we’ve been working on making this experience a reality. The result is CVNAR, a computer vision and augmented reality solution. CVNAR is a powerful software framework containing mathematical algorithms that process a vast amount of road data in real time to generate intuitive prompts and alerts. CVNAR has built-in algorithms for road and pedestrian detection, vehicle recognition and tracking, lane detection, facade recognition and texture extraction, road sign recognition, and parking space search. It performs relative and absolute positioning and easily integrates with navigation, the map database, sensors, and other data sources. A unique feature of CVNAR is its extrapolation engine for latency avoidance.

Detecting and recognizing road signs, pedestrians, traffic lanes, gas stations, and other objects.

CVNAR works perfectly with LCD displays and smartglasses, but it is ultimately built for HUDs. Data from cameras, sensors, CAN, and navigation maps are fused and processed to create an extendable metadata output that describes all augmented objects. It takes a HUD and an eye-tracking camera to implement CVNAR in a vehicle. CVNAR will track the driver’s gaze and adjust the position of the augmented objects in the driver’s line of sight to make sure they don’t obstruct anything important — all in real time.

Alerting the driver to an empty parking spot.

This is not all that CVNAR can do. New car models come packed with infotainment features that take time to learn and memorize. The CVNAR-based smartphone app can help. It turns your smartphone into an interactive guide. Point your phone camera to your dashboard and use augmented prompts to find out more about a particular car function. It can work under the hood, too.

Era of a software-defined car
A modern car runs on code as much as it runs on gasoline (or a battery-powered electric motor). Today, it takes over 100 million lines of software code to get a premium car going, and the amount of software necessary keeps expanding. At Luxoft, we are excited about the car’s digital future, and we work every day to help bring it about, by developing cutting-edge automotive solutions for leading global vehicle manufacturers.

Offering a wide range of embedded software development and integration services for in-vehicle infotainment and telematics systems, digital instrument clusters, and head-up displays, Luxoft has developed User Experience (UX) and Human Machine Interface (HMI) technology for millions of vehicles on the road today. We push the envelope of technology in such areas as situation-aware HMI, computer vision and augmented reality, while Luxoft’s products, the Populus and Teora UX and HMI design tool chains, power the development of award-winning automotive HMIs and slash time to market.

Software holds the key to the future of cars. It is essential to creating a customized user experience in vehicles. With over-the-air updates, software offers unmatched flexibility and scalability. Finally, it takes safety to the next level with its ability to simulate human-like logic through complex algorithms.

You can view Luxoft’s CVNAR solution running on a QNX-based ADAS demo this week at CES, in the BlackBerry booth: LVCC North Hall, #325.

---

About Alex
Alex Leonov has been in the automotive and IT industry for over 18 years in various business development and marketing roles. Currently, Alex leads the global marketing efforts of Luxoft Automotive.

Read More