BlackBerry QNX’s self-driving Lincoln MKZ – what’s under the hood?

Kerry Johnson

Sr. Product Manager

BlackBerry QNX

At CES 2017, BlackBerry-QNX unveiled its self-driving Lincoln MKZ. 

In years past , BlackBerry QNX has become known for displaying its innovative technology in its concept cars, which included infotainment, mobile device connectivity, digital instrument clusters and ADAS. This year BlackBerry QNX has outfitted a Lincoln MKZ to demonstrate a self-driving vehicle. The Lincoln MKZ is much more than a demonstration vehicle – it is an engineering prototype that allows BlackBerry QNX engineers to experiment with and develop new technologies for the autonomous vehicle market.

You may wonder why BlackBerry QNX chose a Lincoln MKZ for its autonomous driving car. The reason is straight forward. The 2017 Lincoln MKZ comes equipped, from the factory, with all the necessary drive-by-wire capabilities. All of the driving systems (throttle, gearbox, steering and braking) can be completely controlled electronically. By using this capability as a starting point, BlackBerry QNX and its partners are able to focus on adding other self-driving capabilities such as the sensors, route planning, and maneuvering.

While providing the foundational software, BlackBerry QNX did not build this self driving vehicle alone. We worked closely with Renesas, University of Waterloo, Polysync and Cogent, to put the car on the road.

The following is a brief walkthrough of the technologies inside the Lincoln MKZ:

BlackBerry QNX

BlackBerry QNX’s goal was to build an autonomous vehicle using commercial embedded processors and safety certified embedded operating system (OS). At the core of the design was QNX’s safety certified OS, which powers all of  the intelligent software modules. QNX’s middleware serves to integrate RADAR, LIDAR sensors , multiple camera inputs and vehicle networking. BlackBerry QNX provided a port of the OpenCV library to help with the vision processing functions delivered by Cogent. 

BlackBerry QNX also provided a port of Robot OS (ROS), so that the University of Waterloo could easily bring their self-driving software algorithms to the car without having to re-write large portions of code.

The ROS software components are not truly embedded, production oriented software. However, in building an autonomous car we chose a phased approach. We chose to use existing software to test and validate the solution. This saves time and allows flexible prototyping. Once the code is finalized we can convert it into an embedded solution.   

University of Waterloo

The University of Waterloo, one of Canada’s leading autonomous driving research institutions, contributed several software components, including static and dynamic environment perception, path planning, maneuvering and dispatching control commands to the various actuators. It should be noted that, at the outset of the project, the University of Waterloo already had a number of these components operational. Part of the activity was to port the software from Linux to QNX – a task made simple by BlackBerry QNX’s support for the POSIX standard.

Polysync

Polysync provided their framework for distributed communications and sensor integration. They also provided system data visualization tools, so the engineers could see how the system was operating from a central console.

Cogent

Cogent provided a number of vision processing algorithms that processed input from multiple camera sensors.

Renesas

The compute horsepower in the Lincoln MKZ comes from two Renesas R-Drive reference boards. Each Renesas R-Drive board has two Renesas R-Car system on chips (SoCs), each with quad-core ARM processors and image processing accelerators. Two R-Drive systems were used so that fail-over scenarios could be tested.

Sensors

The following sensors were used to construct a 360-degree view of the surroundings and to achieve accurate positioning of the car:

• 1 Delphi long range radar
• 1 Delphi short & medium range radar
• 2 Velodyne LIDARs
• 1 forward-facing Point Grey camera
• High precision GPS and IMU (Inertial Management Unit)

The car is now running on a test track.  In the following years BlackBerry QNX will continue to refine the system towards production oriented hardware and software. 

Read More

BlackBerry QNX Partners with Obigo to Build a Better Browser for Your Car

John Wall

Senior Vice-President and Head of QNX Software Systems

 

  

Software plays a big role in current vehicles. The best example is your in-vehicle infotainment system. With just a few taps, you can play music from your mobile device, view and manage advanced navigation systems, make phone calls, tap into traffic reports and weather forecasts, all from your car’s center display console. Drivers and passengers love these systems and are dependent on them. However, Infotainment has its own challenges to address. Security aside, the infotainment system needs to be able to provide support for the latest and greatest web browser technology to give the users access to the content they want the instant they want it, whether on the web or off. You do not, after all, want to be fumbling with an unresponsive display when you are on the road.

At BlackBerry, we are constantly seeking ways to improve the vehicle cockpit experience and to enhance our QNX CAR Platform for Infotainment (QNX CAR). This is why we are proud to announce our partnership with Obigo a leading Korea-based provider of mobile Internet services and browser software. Obigo is working with is to deliver a powerful, Chrome Blink-based HTML5 engine, which will enhance the browser experience of your infotainment systems.

As part of the agreement, our internal HTML5 team will be augmented by Obigo’s team of experts who will help to optimize the latest Blink browser technology with QNX CAR (pictured). QNX CAR, our best in class Infotainment system, has more than 50% global market share and is present in over 60 million vehicles. With QNX bringing this new browser to market, vendors and manufacturers can enrich their driving experience through new applications and services, and drive further customer satisfaction.

“Many automotive OEMs and Tier-1s still view open source HTML browsers as lightweight,” explains Obigo CEO David Hwang. “With almost two decades of browser experience on embedded systems, we have been working to change those perceptions with highly optimized technology that address open source browser performance issues. Working with BlackBerry-QNX, we plan to develop a product that will boost interest in HTML5 technology for emerging in-vehicle applications and services.”

Obigo’s technology will also simplify the coding of HTML 5 browsers for new infotainment systems and accelerate the product development cycle. Obigo joins the ecosystem of BlackBerry QNX partners that collectively offer the best-in-class system level solution for our infotainment customers.

Read More

From Concept to Reality: BlackBerry-QNX's Groundbreaking CES Tradition

Thomas Bloor
Business Development Manager, QNX BlackBerry

The annual Consumer Electronics Show in Las Vegas has been growing in importance for the automotive industry over the years. You can hardly fail to notice that this year, as in previous years, the big automakers vie for floor space and attention with the glut of big screen TVs and other consumer goods. As always, BlacBerry QNX will be in the North Hall, proudly in the middle of the big automotive OEMs. 

At CES BlackBerryQNX has an enviable history of bringing concept cars that rival anything on the show floor, with one important difference – ours are not pure flights of fancy, and we show technologies that will become realities in the near future.

We started this trend back in 2010 with an LTE-connected Toyota Prius – 18 months before the first commercial LTE deployment in mid-2011. Working with Alcatel-Lucent to provide the experimental network, we demonstrated Google maps functionality with local search and an embedded Pandora radio app in a car for the first time. Connectivity is standard in many cars today, but in 2010 we demonstrated the future.

2012 brought us a CNET "Best of CES" award for demonstrating cloud-based natural language voice recognition, text-to-speech, and NFC based one-touch Bluetooth pairing.  Simply touching your phone to an NFC reader in the center console automatically paired the phone and car. 

In 2013 we got ahead of the trend for ever larger center stack displays – with detailed 3D maps and voice recognition Keyword Spotting – common today in smartphones but a first in a car. Simply saying "Hello Bentley" enabled you to start interacting with the natural language cloud based voice recognition Powered by AT&T’s Watson. 

2014 took literally us in a different direction. A 21-inch horizontally orientated center stack display extends across the dash, naturally extending the interaction and functionality towards the passenger.  Behind the screens the instrument cluster was integrated with the center stack running both driver information and IVI functions. With seamless controllability across the touch screen, physical buttons, and the jog wheel controls multi-modal input was highlighted across all available functionality. 

Not content with that, we foreshadowed greater integration of ADAS functionality warnings to the driver. In 2014 we warned the driver if local speed limits were exceeded through both the cluster and verbally through text-to-speech, and we followed this up in 2015 with a system that recommends an appropriate speed for upcoming curves based upon driving conditions and the radius of the bend.

So, what innovations will we be showing in 2017? I’m not allowed to tell you just yet but, in a first (for us), we’ll be showing both future and current production technologies and innovations.

Building on our products ranging from in-car acoustics through our comprehensive QNX-CAR application platform and to next generation driver assistance/autonomous drive we will be demonstrating how technology can enhance the user experience and increase safety for drivers and passengers.

While demonstrating technologies that will come to future production vehicles, these cars are not just "show floor wonders" because our automotive knowledge enables us to build demonstrators for the real world, which can be driven, thus allowing technologies to be experienced first-hand.

Read More

Holistic Security for the Software-Defined Car

Bill Boldt
Sr. Business Development Manager, Security
Blackberry Certicom

Due to high profile hacks on cars, it is hard to argue that without security you can have safety.   So, security is emerging as perhaps the most important factor in the evolution of the connected autonomous car.
 
Cars are the most software intensive systems in the universe with far more lines of code than even a state of the art jet fighter. By being such complex digital systems they have become prime targets for attack, and that is where cryptographic countermeasures come in.

Connecting the dots – in the emerging software-defined world safety increasingly
comes from security and security comes from cryptography. Robust cryptographic security implementation is how you increase trust, and when it comes to a car every system must be
trusted: inside the car, in the smart infrastructure, in emerging applications-based ecosystems, and in the manufacturing supply chain. When considering automotive security,
many factors come into play. Some are noted here:
                       

• Automotive security fundamentally depends on the security of the operating system. For example, a microkernel architecture that separates critical OS components into their own protected memory partitions, provides temporal separation, and provides network security, among other things can greatly reduce the attack surface.

• Security assets (crypto keys, serial numbers, etc.) must be securely installed into electronic devices such as Electronic Control Units (ECUs), domain/area controllers, and other processors. This process is called "personalization".

• Electronic devices will often get personalized and installed into vehicles in globally located factories, which should utilize secure equipment and processes to ensure security of the devices.

•  Devices must be updateable at dealers and repair shops. 

• Aftermarket suppliers must be able to sell and update secure devices, and

• OEMs must be able to authorize or not authorize specific electronic devices at
  manufacturing time and after the car is in use (for example to enforce warrantee policies).
  

And, there are many more.


Personalizing a device such as a networked ECU means that it will become one of a kind. However, by definition that device cannot be used anywhere else. It becomes a unique stock keeping unit (SKU), which is averse to the purpose of flexible, just in time manufacturing flows. Security versus manufacturing flexibility is a serious trade off that will play a part of any automotive security design decision.

Security robustness versus cost is another critical trade off, and applies to the manufacturing infrastructure and the design of the secure systems inside and outside the vehicle. Because security must be injected in the factory and in the field, a secure manufacturing system must have global reach, be manageable on a distributed basis, be updatable by various entities, and remain secure for years. In addition, security updates will increasingly be made over the air, and the systems that do that must by highly secure while being easy to manage. To maintain the maximum amount of flexibility, personalization and updating should be moved as close as possible to the very last minute, which is becoming a critical objective of the global manufacturing blue print. 

Blackberry Brings It All Together

In the car, outside the car, and in the manufacturing supply chain, security must be designed with best practices in mind right from the start, and BlackBerry Professional Services can help with that. BlackBerry QNX provides mission-critical automotive software proven in the automotive market.  QNX software is well known for safety and new products are setting the new standard for security.

BlackBerry's Certicom subsidiary provides certified cryptographic code and design consulting, as well as secure equipment and managed services that harden the automotive supply chain. Completing the picture, BlackBerry's secure OTA managed services make it easy to update software and security assets over the air. When it comes to automotive security, BlackBerry brings it all together.

Read More

Autonomous Cars – Part 3: Technology Consolidation

Kaivan Karimi
SVP of Strategy and Business Development
BlackBerry Technology Solutions

The amount of software in a car is mushrooming with there being over 100 million lines of code in a modern car, which is more than most any other system.

Today cars are controlled via hardware electronic control units (ECUs) running the millions of lines of code.   60 to 100 ECUs are found in most newer cars today and that number is growing.  High end cars can have even more.  Reducing the number of ECUs in favor of reduced number of domain/area controllers is the new trend.  The idea is to reduce the complexities associated with software development, reduce the weight of the car, and reduce the overall cost. It also makes software upgradability less complex, where software functionalities can be enhanced to extend the life of a platform and offer a very large return on investment. 

Another benefit is that software can be more easily upgraded Over-The-Air (OTA) for minor or major fixes, respond to security issues, and provide other enhancements without the need to bring a car to the dealership.  This not only saves time, but also adds to the safety, security, and reliability of the car, while lowering the overall maintenance cost for the vehicle. According to research firm IHS, about 4.6 million cars received OTA software updates for telematics applications last year, and by the year 2022 forty-three million cars are expected to be using OTA services.  That is clearly a huge increase.

Some of the other technology components of Advanced Driver Assistance Systems (ADAS) are noted below:

Maps

While most people do not consider maps as a component of an ADAS system, in the future they will play a key role in assisting drivers to operate vehicles safely and adapt to driving changes based on location, such as changing what side of the road you drive on when you hit a border crossing. Maps provide a necessary input to augment the information that is provided by the various sensors in the car. This is not just macro-level geological data for finding directions, but also for augmenting functions such as camera-based traffic sign and roadway information detection, as well as infrastructure information. Cloud based processing will then be used to integrate the data sent by all vehicles into a global map that gets updated cooperatively by all drivers, including the road pot-holes to avoid, new roadway signs added, or rerouting due to construction.

Sensor Fusion

Sensor fusion means combining information and data from different sensors, leveraging the individual advantages of each sensor to complement and cover the weaknesses other sensors. The whole is greater than the sum of the parts, which means the individual sensors’ functions. This is very similar to what our brain does. You do not need to touch a pot of boiling water to know it is very hot, because your eyes to see the bubbling water and the steam on the top of the pot. In an ADAS system, the same thing happens: The sensor inputs are fused together for the ADAS domain controller to formulate a conclusive opinion about an event with better situational awareness, rather than just relying on a certain sensor’s data individually. This notion is at the heart of how any robot operates, but is especially important with the mission critical functionalities needed by connected autonomous cars.

HW & SW Roadmap to Consolidation

As the modern CPU increases in processing power, and decreases in electrical power consumption due to smaller process geometries, it would lead one to believe that consolidating multiple ECU functions onto one physical processor may result in significant cost savings. While that is true, consolidation needs to be balanced with a few important factors:

1.  The increase in leakage current as semiconductor process geometries get smaller (this is a downside of Moore’s Law) 
2. Thermal issues increase as clock speeds increase 
3. The extent to which the software can be multithreaded to take advantage of new multi-core  processors.

The auto industry will be going through a transformation with ECU consolidation into single powerful multi-core processors that is similar to what happened in early 2000s in the networking industry.  At that time I had a front seat to the networking debate as I was driving some products a large semiconductor company. What happened was that most network and baseband processor semiconductor suppliers for both wired and wireless infrastructure business moved from single to dual to quad-core processors.  I remember a day when people were planning to pack as many as 80 cores into a single chip.

There is a huge difference between the software requirements for mutli-core processing in the networking and automotive industries.  

The elephant in the automotive room is the need to combine mission-critical with non-mission critical functionalities into the same processor, while separating and isolating these functions effectively from each other from a safety and security perspective. This single fundamental requirement becomes the basis for what types of software framework and architecture needs to be used.

Multi-core Processing

At a very high-level, all multi-core processors pack multiple processing units (cores) into a single    physical package—just like it sounds. But, this is where the similarities end. Other architectural factors come into play and determine the application fit, throughput, bandwidth, effective horsepower, and software architectures suitable for an optimal processing environment. Some of the considerations are noted below:

• Choice and configuration of interconnect buses and shared memory schemes

• Choice of homogeneous multi-core systems with identical cores sharing the same instruction sets, vs. heterogeneous multi-core systems with identical cores (some with same instruction set, and some with different ones 

• Heterogeneous multi-core systems that mix different types of processor cores for application specific use cases (e.g. mix of MPUs, DSPs, GPUs, etc.).

• Mix of the above core with localized memories and predefined high-level functions such as micro-coded engines and vector processors

• Mix of cores and architectures that allow control and data path processing in a single core for communication applications 

• Choice of architectural implementations such as VLIW, vector or multithread processors, fine-grain vs. coarse grain processors, etc.

The improvement in performance by using multi-core processors can only happen if the software running on the processor can take advantage of every last cycle that the multiple core device can offer. It also assumes that the interconnect buses and interfaces between the cores and the world outside of the chip, as well as between the cores, and the interaction between the cores and the memory architecture are properly modeled and designed for the end application, so that there are no design bottle necks introduced. 

This situation is analogous to adding multiple streets and multiple lanes in and out of a parking lot. If the electronic door to go in and out of that parking lot is too slow to accommodate the extra traffic, you will cause bad congestion, and the traffic throughput in and out of the parking lot would be as good as the speed of that electronic door. You may need to open the gate altogether, but have a traffic cop that coordinates the flow of traffic in and out of different entrances, into different parking spots. That is exactly what you would also need in the world of software, namely a traffic cop for the processes running in the given multi-core architecture. That is where a hypervisor comes in, which is to act as that traffic cop.

QNX offers a hypervisor and other safety- and mission-critical software for make connected autonomous cars safe reliable, secure, and trusted.

The next blog will address the hypervisor/traffic cop, and describe how they make the software-defined future more autonomous and safe.

                                                  

Kaivan Karimi is the SVP of Strategy and Business Development at BlackBerry Technology Solutions (BTS). His responsibilities include operationalizing growth strategies, product marketing and business development, eco-system enablement, and execution of business priorities. He has been an IoT evangelist since 2010, bringing more than two decades of experience working in cellular, connectivity, networking, sensors, and microcontroller semiconductor markets. Kaivan holds graduate degrees in engineering (MSEE) and business (MBA). Prior to joining BlackBerry, he was the VP and General Manager of Atmel's wireless MCUs and IOT business unit.

Read More