-EMAINLINE

A first photo on mainline Linux

Yassine Oudjana — Wed, 24 Aug 2022 00:00:00 +0000

Cameras on phones have gotten so good that they are often taken for granted. They are expected to always work and produce good quality photos with fairly accurate colors. Those trying to make them work on a Linux phone for the first time would not have it as easy though, as they get to discover the several components involved in long complicated processes to arrive at a ready-to-use JPEG image. This is why cameras remain one of, if not the least supported feature of all Linux phones, whether originally made to run Linux or had it ported to them.

To find out how it all works, an attempt is made to enable the main camera on a Xiaomi Mi Note 2 on mainline Linux.

Understanding the hardware

What the average end-user might know about this phone is that it has a Sony IMX318 main camera with a native resolution of 5488x4112 adding up to 22.56 MP. Some might also know that the SoC on the phone, the Qualcomm Snapdragon 821, has an ISP – short for Image Signal Processor – that helps with processing photos. To enable the camera on Linux for the first time would require a much lower-level understanding of the hardware, however. A quick look at the schematics of this phone would be a good way to begin.

Searching for “camera” shows a few interesting blocks. This one labeled REAR CAMERA seems like a good starting point: This show several lines coming out of a socket, which the camera module presumably plugs into. Starting with the obvious, there are a few lines going up. These are rails that supply power to the camera module. CAM_MCLK0 must be a clock, and CAM1_RST_N a reset line. As for data, two categories of lines can be seen: One with the CCI_I2C prefix, and another with MIPI_CSI. CCI, or Camera Control Interface, is just an I²C bus dedicated to controlling the camera module. Since I²C has quite a low bandwidth, it would not be adequate to carry image data from the sensor, hence the use of the MIPI Camera Serial Interface. MIPI CSI is a high bandwidth serial bus designed specifically for transferring image data from cameras.

Another block labeled FRONT CAMERA is also shown in the schematic diagram: It shows a similar setup, with some power rails, a clock line, a reset line, as well as CCI and CSI lines.

A closer look at the camera module

A camera module consists of an image sensor that collects light and outputs signals to recreate the image digitally, as well as a lens to focus light falling on the sensor. Depending on the camera module, there can also be an actuator that physically moves the lens to change the focus distance. Many camera modules found on phones these days have multiple sets of sensors and lenses with different focal lengths for added flexibility. The main camera on the Mi Note 2 – as mentioned previously – has a Sony IMX318 image sensor. This camera can also change focus distance both automatically and manually, so it must have a lens actuator. The image sensor is the most important part of the operation, so it will be the focus target*.

*pun intended.

Documentation would become necessary to learn more about the image sensor. To write a driver for the sensor would of course require at least a register map with definitions, which is sometimes included in the datasheet, or found in a separate Technical Reference Manual or Functional Specification. Unfortunately, there are no publicly available documents to be found, other than a product description including general specifications of the sensor.

With this phone originally running Android, the downstream kernel can be taken as an alternative source of information. In this case however, the vendor decided to use a proprietary userspace driver, meaning there is no source code showing how the sensor is controlled.

Dead end?

Having no documentation, will the camera require yet another reverse engineering effort? We must go deeper.

Having a proprietary driver for the camera on this phone does not necessarily mean that it is done similarly in other phones, so there might be an open source kernel driver in another downstream kernel for a different device. Searching for “IMX318” across GitHub reveals a couple of drivers, one belonging to a NVIDIA Jetson Nano kernel, and the other to some common MediaTek MT6757 kernel.

Who knew that two of the least open source-friendly vendors would become the most helpful one day...

Before looking at the downstream drivers however, it would be better to take a look at an existing mainline driver for a similar sensor to gain a general understanding of how these drivers are constructed, and figure out what needs to be known to write a new one. I²C-controlled camera sensor drivers are located in drivers/media/i2c. Here, several drivers for Sony sensors are found, distingushed by the imx prefix in their names.

A look at an existing driver

Almost every Linux driver starts at a probe function where all necessary initialization steps are made to prepare the hardware for operation, as well as to register the driver to the relevant kernel subsystems. Looking at the IMX214 driver, the general flow of events goes like this:

The device tree node is first parsed, endpoints are identified and link frequency compatibility is tested:

  ret = imx214_parse_fwnode(dev);
      if (ret)
          return ret;

  static int imx214_parse_fwnode(struct device *dev)
  {
      struct fwnode_handle *endpoint;
      struct v4l2_fwnode_endpoint bus_cfg = {
          .bus_type = V4L2_MBUS_CSI2_DPHY,
      };
      unsigned int i;
      int ret;

      endpoint = fwnode_graph_get_next_endpoint(dev_fwnode(dev), NULL);
      if (!endpoint) {
          dev_err(dev, "endpoint node not found\n");
          return -EINVAL;
      }

      ret = v4l2_fwnode_endpoint_alloc_parse(endpoint, &bus_cfg);
      if (ret) {
          dev_err(dev, "parsing endpoint node failed\n");
          goto done;
      }

      for (i = 0; i < bus_cfg.nr_of_link_frequencies; i++)
          if (bus_cfg.link_frequencies[i] == IMX214_DEFAULT_LINK_FREQ)
              break;

      if (i == bus_cfg.nr_of_link_frequencies) {
          dev_err(dev, "link-frequencies %d not supported, Please review your DT\n",
              IMX214_DEFAULT_LINK_FREQ);
          ret = -EINVAL;
          goto done;
      }

  done:
      v4l2_fwnode_endpoint_free(&bus_cfg);
      fwnode_handle_put(endpoint);
      return ret;
  }

The external clock is located and configured:

  imx214->xclk = devm_clk_get(dev, NULL);
  if (IS_ERR(imx214->xclk)) {
      dev_err(dev, "could not get xclk");
      return PTR_ERR(imx214->xclk);
  }

  ret = clk_set_rate(imx214->xclk, IMX214_DEFAULT_CLK_FREQ);
  if (ret) {
      dev_err(dev, "could not set xclk frequency\n");
      return ret;
  }

Regulators and GPIOs are parsed:

  ret = imx214_get_regulators(dev, imx214);
  if (ret < 0) {
      dev_err(dev, "cannot get regulators\n");
      return ret;
  }

  imx214->enable_gpio = devm_gpiod_get(dev, "enable", GPIOD_OUT_LOW);
  if (IS_ERR(imx214->enable_gpio)) {
      dev_err(dev, "cannot get enable gpio\n");
      return PTR_ERR(imx214->enable_gpio);
  }

A V4L2 subdevice is initialized,

  v4l2_i2c_subdev_init(&imx214->sd, client, &imx214_subdev_ops);

and its controls are configured:

  v4l2_ctrl_handler_init(&imx214->ctrls, 3);

  imx214->pixel_rate = v4l2_ctrl_new_std(&imx214->ctrls, NULL,
                         V4L2_CID_PIXEL_RATE, 0,
                         IMX214_DEFAULT_PIXEL_RATE, 1,
                         IMX214_DEFAULT_PIXEL_RATE);
  imx214->link_freq = v4l2_ctrl_new_int_menu(&imx214->ctrls, NULL,
                         V4L2_CID_LINK_FREQ,
                         ARRAY_SIZE(link_freq) - 1,
                         0, link_freq);
  if (imx214->link_freq)
      imx214->link_freq->flags |= V4L2_CTRL_FLAG_READ_ONLY;

  imx214->exposure = v4l2_ctrl_new_std(&imx214->ctrls, &imx214_ctrl_ops,
                       V4L2_CID_EXPOSURE,
                       0, 3184, 1, 0x0c70);

  imx214->unit_size = v4l2_ctrl_new_std_compound(&imx214->ctrls,
              NULL,
              V4L2_CID_UNIT_CELL_SIZE,
              v4l2_ctrl_ptr_create((void *)&unit_size));
  ret = imx214->ctrls.error;
  if (ret) {
      dev_err(&client->dev, "%s control init failed (%d)\n",
          __func__, ret);
      goto free_ctrl;
  }

  imx214->sd.ctrl_handler = &imx214->ctrls;
  mutex_init(&imx214->mutex);
  imx214->ctrls.lock = &imx214->mutex;

Then finally a media entity is configured and the V4L2 subdevice is registered:

  imx214->sd.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
  imx214->pad.flags = MEDIA_PAD_FL_SOURCE;
  imx214->sd.dev = &client->dev;
  imx214->sd.entity.function = MEDIA_ENT_F_CAM_SENSOR;

  ret = media_entity_pads_init(&imx214->sd.entity, 1, &imx214->pad);
  if (ret < 0) {
      dev_err(dev, "could not register media entity\n");
      goto free_ctrl;
  }

  imx214_entity_init_cfg(&imx214->sd, NULL);

  ret = v4l2_async_register_subdev_sensor(&imx214->sd);
  if (ret < 0) {
      dev_err(dev, "could not register v4l2 device\n");
      goto free_entity;
  }

Infrastructure

These steps show some basic elements the sensor needs, namely clocks and regulators. The sensor takes in one clock signal and uses it internally to generate all clock signals it needs to function. This is shown as xclk (short for external clock) in the driver. A second clock that is part of the MIPI CSI bus is used to synchronize the CSI links, the frequency of which is referred to as link frequency in the driver. The frequency read from the device tree must be supported by the sensor, hence the compatibility checking. Link frequency will become important later.

As for power, the sensor needs three supplies: one to power the digital part of the IC, one to power its analog part, and one to pull-up I/O lines.

V4L2

Short for Video for Linux version 2, V4L2 is a Linux API for managing video devices of all kinds, including codecs, TV tuners, multiplexers and of course, cameras. It allows for dealing with complicated devices as sets of simpler subdevices, each having its own controls, as well as source and sink pads to link it with other subdevices and control the flow of data through a pipeline. The kernel side of V4L2 is used throughout the sensor driver.

Back to the IMX318

The first thing to do would be to power on the sensor. To do so, the power rails it is hooked up to must be brought up, the external clock must be enabled and set to a suitable frequency, and the reset line must be asserted then deasserted to reset the sensor. It might seem simple at first, but as it would turn out later, it needs to be done in a specific way in order to get the sensor into a working state.

A good first step to find the correct power on sequence would be to see how the stock driver does it. Although the vendor decided to use a proprietary blob to drive the sensor, the kernel remains open-source, and anything regarding clocks, regulators or GPIOs goes through it at some point. Conveniently, they decided to have the userspace driver give the kernel instructions on a power sequence which it would then execute. Even more so, the parsing and execution of the instructions happens in a single function named msm_camera_power_up. It even has dynamic debug messages that if enabled, would allow for monitoring of the power on process as it happens. This function calls msm_cam_sensor_handle_reg_gpio for enabling regulators, so debug messages must be enabled there too. Once all relevant messages are enabled, the sequence can be seen in the kernel log:

[   12.194390] msm_camera_power_up:1538
[   12.194989] msm_camera_power_up index 0
[   12.194994] msm_camera_power_up type 1
[   12.194998] msm_camera_power_up:1607 gpio set 86 val 0
[   12.196751] msm_camera_power_up index 1
[   12.196758] msm_camera_power_up type 1
[   12.196761] msm_camera_power_up:1607 gpio set 30 val 0
[   12.198801] msm_camera_power_up index 2
[   12.198808] msm_camera_power_up type 2
[   12.199424] msm_cam_sensor_handle_reg_gpio: 1420 Seq val: 2, config: 1
[   12.199429] msm_cam_sensor_handle_reg_gpio: 1454 GPIO offset: 4, seq_val: 2
[   12.199433] msm_camera_power_up index 3
[   12.199435] msm_camera_power_up type 2
[   12.199888] msm_cam_sensor_handle_reg_gpio: 1420 Seq val: 0, config: 1
[   12.199892] msm_cam_sensor_handle_reg_gpio: 1454 GPIO offset: 5, seq_val: 0
[   12.199895] msm_camera_power_up index 4
[   12.199898] msm_camera_power_up type 2
[   12.201948] msm_cam_sensor_handle_reg_gpio: 1420 Seq val: 1, config: 1
[   12.201954] msm_cam_sensor_handle_reg_gpio: 1454 GPIO offset: 3, seq_val: 1
[   12.201960] msm_camera_power_up index 5
[   12.201963] msm_camera_power_up type 2
[   12.202577] msm_cam_sensor_handle_reg_gpio: 1420 Seq val: 3, config: 1
[   12.202582] msm_cam_sensor_handle_reg_gpio: 1454 GPIO offset: 6, seq_val: 3
[   12.202587] msm_camera_power_up index 6
[   12.202590] msm_camera_power_up type 0
[   12.203887] msm_camera_power_up index 7
[   12.203894] msm_camera_power_up type 1
[   12.203898] msm_camera_power_up:1607 gpio set 86 val 2
[   12.205735] MSM-CPP cpp_init_hardware:1011 CPP HW Version: 0x60000000
[   12.205745] MSM-CPP cpp_init_hardware:1029 stream_cnt:0
[   12.214102] msm_camera_power_up index 8
[   12.214109] msm_camera_power_up type 1
[   12.214113] msm_camera_power_up:1607 gpio set 30 val 2
[   12.219075] msm_pm_qos_update_request: update request -1
[   12.219080] msm_pm_qos_add_request: add request
[   12.225387] msm_cci_init:1434: hw_version = 0x10040000
[   12.225466] msm_camera_power_up exit

This might look like a lot at first glance, but it basically does the following:

Set GPIO 86 to LOW,
Set GPIO 30 to LOW,
Enable V_ANA supply,
Enable V_DIG supply,
Enable V_I/O supply,
Enable V_AF supply,
Enable external clock,
Set GPIO 86 to HIGH,
Set GPIO 30 to HIGH.

Supplies are found using the GPIO offset values shown in the msm_cam_sensor_handle_reg_gpio messages, which correspond to entries in an enum defined in a separate header file.

Looking at schematics, GPIO 30 is found to be the CAM1_RST_N reset pin seen before: The _N suffix suggests that it is active low, which is to say that it is asserted by setting it to LOW, and deasserted by setting it to HIGH.

As for GPIO 86, its name suggests that it is an enable pin of a regulator: Following it, we end up at what seems to be a LDO (short for Low-Dropout Regulator):

With the GPIOs figured out, now come regulators, which can be found in downstream DTS:

cam_vdig-supply = <&pm8994_s3>;
cam_vio-supply = <&pm8994_lvs1>;
cam_vana-supply = <&pm8994_l17>;
cam_vmipi-supply = <&pm8994_l11>;
qcom,cam-vreg-name = "cam_vdig", "cam_vio", "cam_vana", "cam_vmipi";

V_AF supply can be found separately in the actuator node

cam_vaf-supply = <&pm8994_l23>;
qcom,cam-vreg-name = "cam_vaf";

There does not seem to be any references to the LDO found in schematics earlier. However, when looking closely, it can be noticed that V_DIG supply is the same as V_IN of the LDO. This would suggest that V_DIG is actually the LDO output rather than S3 of the PM8994 PMIC. In fact, this pattern is frequently seen in the downstream kernel, where an LDO (or any kind of external regulator with fixed voltage output and an enable pin) is not described in DTS, and its enable pin is treated as if it were of the component the regulator supplies power to.

Finally, the external clock and its frequency are also found in downstream DTS:

clocks = <&clock_mmss clk_mclk0_clk_src>,
		<&clock_mmss clk_camss_mclk0_clk>;
clock-names = "cam_src_clk", "cam_clk";
qcom,clock-rates = <24000000 0>;

Putting all of this together, the power sequence would look like this:

Disable V_DIG supply,
Assert reset
Enable V_ANA supply,
Enable parent supply of V_DIG supply,
Enable V_I/O supply,
Enable V_AF supply,
Enable external clock and set it to 24MHz,
Enable V_DIG supply,
Deassert reset.

This sequence would prove to be ineffective, however. Fortunately, there is source code to refer to in this case: The driver from the Jetson Nano kernel. This driver’s power on function suggests adding a 19ms delay after deasserting reset.

There was also a mistake I made in the way I described the external clock in DTS, which might have had an even bigger hand at preventing the sensor from powering up at first. The reason I made it to begin with is a bit complicated and shouldn't be an issue for anyone else, so I will not talk about it for brevity's sake...

Combining information from the sequence captured on the running downstream kernel and the driver in the Jetson Nano kernel tree, and after some experimentation, a working power on function is made:

static int imx318_power_on(struct device *dev)
{
	struct i2c_client *client = to_i2c_client(dev);
	struct v4l2_subdev *sd = i2c_get_clientdata(client);
	struct imx318 *imx318 = to_imx318(sd);
	int ret;

	gpiod_set_value_cansleep(imx318->reset_gpio, 1);

	ret = regulator_bulk_enable(IMX318_NUM_SUPPLIES, imx318->supplies);
	if (ret) {
		dev_err(imx318->dev, "Failed to enable regulators: %d\n", ret);
		regulator_bulk_disable(IMX318_NUM_SUPPLIES, imx318->supplies);
		return ret;
	}

	ret = clk_prepare_enable(imx318->extclk);
	if (ret < 0) {
		dev_err(imx318->dev, "Failed to enable external clock: %d\n", ret);
		return ret;
	}

	gpiod_set_value_cansleep(imx318->reset_gpio, 0);

	/* Wait for sensor to power on */
	usleep_range(19000, 20000);

	dev_dbg(imx318->dev, "Powered on\n");

	return ret;
}

Sensor initialization

The sensor has to be initialized first before it can capture any images. There are many registers in the sensor that are written to during initialization, some of which are related to the internal clock tree, while others configure image properties, timing, and so on.

Ideally, each of those registers should be labeled clearly in the driver, then written to individually with also clearly labeled values, or if a bitmask, constructed at the time of writing. However, due to lack of documentation, and sometimes simply due to the sheer number of registers that need to be written, many sensor drivers just have a bunch of magic numbers arranged in tables of register-value pairs, which then get applied as needed. Going back to the downstream drivers found previously, such register tables can be seen. In the Jetson Nano driver, the tables are placed in a separate header file, while in the MT6757 driver, they are placed in the source file of the driver.

In both drivers, there is a common table that is quite long, and a few smaller mode tables. The common table is used once for initialization, while the mode tables are used to switch between, well, different modes.

Modes?

While the IMX318 sensor has a native resolution of 5488x4112, it can also operate at other resolutions with various refresh rates. The combination of resolution and refresh rate* is referred to as a mode.

*While other aspects such as formats (more on that later) and crop factors can differ between modes, we will stick to resolutions and refresh rates only to keep things simple.

Due to the difference in bits per image and the number of images that need to be transmitted per second for each mode, The rate at which the sensor has to transmit signals depends on the mode configured. This is where link frequency comes in. Theoretically speaking, the minimum link frequency of a MIPI CSI bus required at a certain mode can be calculated as follows:

\[f = {whbr \over 2n}\]

Where:

\(f\): Link frequency;
\(w\): Image width;
\(h\): Image height;
\(b\): Bit depth (bits per pixel);
\(r\): Refresh rate; and
\(n\): CSI link count.

The MIPI CSI bus operates with double data rate (or DDR for short). This means that two bits are transferred every clock cycle, one on the rising edge, and another on the falling edge:

Furthermore, the bus may consist of multiple links, each transmitting bits separately. These two facts make us arrive at the denominator \(2n\) shown in the equation above, meaning two bits per link transferred in a clock cycle. Meanwhile, the numerator is the number of bits that need to be transferred every second: The number of pixels in an image (\(w*h\)) times the number of bits per pixel (\(b\)), multiplied by the number of images transferred each second (\(r\)).

Alright, math class is over. Back to the fun stuff :)

Formats

An image can be digitally represented in countless ways. The method used to arrange image data into a sequence of bytes is known as an image format. Before looking into formats, it would help to understand how image sensors are constructed.

To put simply, image sensors – like displays – have pixels, lots of pixels, in a grid array. While a display emits light from its pixels, image sensors detect it. Displays usually have subpixels within each pixel, each emitting one of the additive primary colors (red - green - blue). However, image sensors do not have subpixels, but rather a different color filter on each pixel in a certain pattern known as a Bayer filter. While this makes every pixel only sensitive* to one of the primary colors, it is possible to achieve a full color image by interpolating color values from neighboring pixels for each pixel.

*mostly sensitive, but that is not important right now.

The Sony IMX318 sensor, like many other Sony image sensors used on phones, has a 10-bit RGGB bayer format. This means that each pixel has a 10-bit value, and that a 2x2 grid has red, green, green then blue pixels left to right, top to bottom:

Coming back to the bit depth, one might wonder how pixel values are stored in bytes, considering that each pixel has 10 bits while a byte only has 8. The easiest way would be to just store each pixel value in two bytes (numbers represent pixel indices, X represents unused bits):

00000000 00XXXXXX 11111111 11XXXXXX 22222222 22XXXXXX 33333333 33XXXXXX

This is known as an unpacked 10-bit format. This method is quite inefficient though; as it leaves many bits unused. Due to that, it is rarely used outside of processing stages, where the data would likely need to be unpacked anyway.

A much efficient method to handle 10-bit pixel values is to use a packed 10-bit format, in which the bits from 4 pixels (\(4 * 10 = 40\)) fit perfectly into 5 bytes (\(5 * 8 = 40\)). It may sound logical at first to pack the bits sequencially, where bits in a row of 4 pixels would be placed in sequence like this:

00000000 00111111 11112222 22222233 33333333

But as it turns out, the bits are usually packed like this:

00000000 11111111 22222222 33333333 00112233

While this may look strange, it is actually a clever way of packing 10-bit pixel values into bytes. In this arrangement, the eight most significant bits of the values are stored in the first 4 bytes, while the two least significant bits of every value are put together in the fifth byte. This allows for converting a 10-bit image into an 8-bit one by simply discarding every fifth byte in the sequence, leaving only the most significant bits behind. This feature will help with decoding raw sensor data later.

Putting everything together

With all necessary elements found, it is now possible to assemble a functional driver for the IMX318 sensor.

The driver is quite big and has many boring details mostly taken from other similar drivers, so I decided not to explain it in detail. Feel free to go through it here.

Device tree

To bring the driver to life, it is necessary to describe the camera hardware in the device tree.

This was done throughout the making of the driver, including testing the power on sequence shown earlier, but I decided to leave it all until the end to limit confusion.

To begin with, the camera subsystem is enabled, and its analog supply is specified:

&camss {
	status = "okay";

	vdda-supply = <&vreg_l2a_1p25>;
};

The analog supply can be found throughout the downstream device tree in the qcom,mipi-csi-vdd-supply property.

The CCI block of the camera subsystem is also enabled:

&cci {
	status = "okay";
};

A node for the V_DIG supply LDO is added under the root node:

/ {
	vdd_cam_1p1: vdd-cam-1p1 {
		compatible = "regulator-fixed";
		regulator-name = "vdd_cam_1p1";
		regulator-min-microvolt = <1100000>;
		regulator-max-microvolt = <1100000>;
		vin-supply = <&vreg_s3a_1p3>;
		regulator-allow-set-load;

		gpio = <&tlmm 86 0>;
		enable-active-high;
		pinctrl-names = "default";
		pinctrl-0 = <&vdd_cam_1p1_default>;
	};
};

Along with a pin state for its enable pin in the tlmm node (GPIO controller of the SoC):

&tlmm {
	vdd_cam_1p1_default: vdd_cam_1p1_default {
		pins = "gpio86";
		function = "gpio";
		drive-strength = <16>;
		bias-disable;
	};
};

Now, the sensor can be added to the node of the CCI bus it is connected to:

&cci_i2c0 {
	clock-frequency = <1000000>;

	/* Rear camera */
	camera-sensor@1a {
		compatible = "sony,imx318";
		reg = <0x1a>;

		vana-supply = <&vreg_l17a_2p8>;
		vio-supply = <&vreg_lvs1a_1p8>;
		vmipi-supply = <&vreg_l11a_1p1>;
		vdig-supply = <&vdd_cam_1p1>;

		reset-gpios = <&tlmm 30 GPIO_ACTIVE_LOW>;

		clocks = <&mmcc CAMSS_MCLK0_CLK>;
		clock-frequency = <24000000>;
	};
};

Here, all needed supplies, clocks, GPIOs are listed, all of which were found previously. The clock frequency of the CCI bus is also configured. The address of the sensor (as seen in the node name and the reg property) is found by setting a random address at first then running i2cdetect on this CCI bus after the driver is probed and the power on sequence is executed.

Pin states for the reset and clock pins of the sensor are also added to tlmm node:

&tlmm {
	cam_sensor_rear_default: cam_sensor_rear_default {
		pins = "gpio30";
		function = "gpio";
		drive-strength = <2>;
		bias-disable;
	};

	cam_sensor_rear_sleep: cam_sensor_rear_sleep {
		pins = "gpio30";
		function = "gpio";
		drive-strength = <2>;
		bias-pull-up;
	};

	mclk0_default: mclk0_default {
		pins = "gpio13";
		function = "cam_mclk";
		drive-strength = <2>;
		bias-disable;
	};

	mclk0_sleep: mclk0_sleep {
		pins = "gpio13";
		function = "gpio";
		drive-strength = <2>;
		bias-pull-down;
	};
};

function = "cam_mclk" is needed for GPIO 13 to make it output a clock signal for the camera rather than functioning as a regular GPIO.

So far, the camera subsystem has been enabled and the image sensor added, but they have not been linked together. To do so, endpoints – as seen before in the IMX214 driver – are used. An endpoint is described in the sensor node:

&cci_i2c0 {
	camera-sensor@1a {
		port {
			imx318_ep: endpoint {
				data-lanes = <0 1 2 3>;
				link-frequencies = /bits/ 64 <211562400 693600000 846249600>;
				remote-endpoint = <&csiphy0_ep>;
			};
		};
	};
};

Here, CSI data lanes used and supported link frequencies are specified. The equation shown previously is used to calculate those frequencies based on modes supported by the driver. Finally, the endpoint at the other end of the communication channel is specified. In this case, the CSI physical layer lies at the other end of the CSI bus.

Likewise, an endpoint is descibed in the camera subsystem node:

&camss {
	ports {
		port@0 {
			reg = <0>;
			csiphy0_ep: endpoint {
				clock-lanes = <7>;
				data-lanes = <0 1 2 3>;
				remote-endpoint = <&imx318_ep>;
			};
		};
	};
};

Used data lanes are also specified here, as well as the physical index of the clock lane. The remote-endpoint property is used here too, this time pointing towards the sensor endpoint.

With that done, it is time to put the image sensor to use.

Getting data out of the sensor

Now that we have a driver, the sensor can be instructed to start capturing images (or frames) and transmitting them over the MIPI CSI bus. However, we still cannot get the data it outputs; as it has to get through the SoC camera pipeline.

The MSM8996 SoC has a camera subsystem (abbreviated to CAMSS as seen before) with several stages, starting at the CSI physical layer (CSIPHY) which brings in electrical signals from outside the SoC, going through the CSI decoder (CSID) which decodes data carried in those signals, then the ISP interface (ISPIF) which provides access to the ISP for processing, arriving finally at the video frontend (VFE) at which image data is exported from the camera subsystem to other parts of the SoC by pixel interface (PIX) and raw dump interface (RDI) blocks. A great blog post by Linaro, as well as the Linux Kernel Documentation explain this in more detail.

In order to retrieve data from the sensor, all of these stages must be configured and linked together. The media-ctl tool can be used for that. First, the stages are linked together:

# media-ctl -d /dev/media0 -l '"msm_csiphy0":1->"msm_csid0":0[1],"msm_csid0":1->"msm_ispif0":0[1],"msm_ispif0":1->"msm_vfe0_rdi0":0[1]'

A visual representation can be obtained with the same tool, with solid lines showing established links, and dashed lines showing possible ones:

$ media-ctl --print-dot | dot -Tpng

Then, source and sink pads of all stages are configured to a certain image format:

# media-ctl -d /dev/media0 -V '"imx318 3-001a":0[fmt:SRGGB10_1X10/3840x2160 field:none],"msm_csiphy0":0[fmt:SRGGB10_1X10/3840x2160 field:none],"msm_csid0":0[fmt:SRGGB10_1X10/3840x2160 field:none],"msm_ispif0":0[fmt:SRGGB10_1X10/3840x2160 field:none],"msm_vfe0_rdi0":0[fmt:SRGGB10_1X10/3840x2160 field:none]'

Now the pipeline is ready to start streaming, which is to say begin capturing frames from the sensor and passing them through the rest of the pipeline until they reach memory where they can be accessed. v4l2-ctl is used to capture a single frame:

# v4l2-ctl -d /dev/video0 --set-fmt-video=width=3840,height=2160,pixelformat='pRAA' --stream-mmap --stream-to=img --stream-count=1 --verbose

VIDIOC_QUERYCAP: ok
VIDIOC_G_FMT: ok
VIDIOC_S_FMT: ok
Format Video Capture Multiplanar:
	Width/Height      : 3840/2160
	Pixel Format      : 'pRAA' (10-bit Bayer RGRG/GBGB Packed)
	Field             : None
	Number of planes  : 1
	Flags             : 
	Colorspace        : sRGB
	Transfer Function : sRGB
	YCbCr/HSV Encoding: ITU-R 601
	Quantization      : Full Range
	Plane 0           :
	   Bytes per Line : 4800
	   Size Image     : 10368000
		VIDIOC_REQBUFS returned 0 (Success)
		VIDIOC_QUERYBUF returned 0 (Success)
		VIDIOC_QUERYBUF returned 0 (Success)
		VIDIOC_QUERYBUF returned 0 (Success)
		VIDIOC_QUERYBUF returned 0 (Success)
		VIDIOC_QBUF returned 0 (Success)
		VIDIOC_QBUF returned 0 (Success)
		VIDIOC_QBUF returned 0 (Success)
		VIDIOC_QBUF returned 0 (Success)
		VIDIOC_STREAMON returned 0 (Success)

Now we have a file named img that contains raw data captured from the sensor.

Processing data into a viewable image

To get an actual image would require processing the captured sensor data first. The easy part would be to convert pixels into 8-bit depth thanks to the packed 10-bit format. However, due to the bayer filter used on the sensor, getting proper color values at capture resolution would require interpolating values of several pixels in a process called demosaicing. A much easier way to get an image however would be to treat every 4 pixels in a 2x2 square grid as subpixels of a single bigger pixel. This will effectively result in an image with 1/4 the capture resolution, but for testing purposes that would still be more than enough. Putting all of this in a Python script:

import sys
from PIL import Image

width = 3840
height = 2160
# Each group of 4 pixels is 5 bytes long
line_length = width * 5 // 4

f = open(sys.argv[1], "rb")
data = f.read()

# Preview image will have 1/2 the dimensions, 1/4 the resolution
preview_img = Image.new("RGB", (width // 2, height // 2))
preview_pixels = preview_img.load()

x = 0
y = 0
pixel = [0, 0, 0]

for y in range(height // 2):
	for x in range(width // 2):
		# Index of top-left corner of pixel in original array
		i = x * 2
		# Skip 5th bytes to get 8-bit pixels
		i += (i+1) // 4

		# Red
		pixel[0] = data[(y*2) * line_length + i]

		# Green
		pixel[1] = data[(y*2) * line_length + i+1]
		# Average value of second green pixel with first one
		pixel[1] += data[(y*2+1) * line_length + i]
		pixel[1] //= 2

		# Blue
		pixel[2] = data[(y*2+1) * line_length + i+1]

		preview_pixels[x, y] = (pixel[0], pixel[1], pixel[2])

preview_img.save("img.png")

And finally, after some manual adjustments:

The Mi Note 2 is starting to open its eye...

First image ever taken while running the mainline Linux kernel, with a fully open-source camera stack from the camera sensor and SoC camera subsystem drivers to the userspace tools. pic.twitter.com/JfB1P8yrK2
— Yassine Oudjana (@y_oudjana) December 16, 2021

Next steps

While getting the camera sensor to work on mainline Linux at all is a great achievement, it is quite useless if the images it captures are underexposed, have inaccurate colors and are out of focus. In order to achieve proper camera support, at least a few things must be taken care of:

Gain and exposure controls must be added to the sensor driver. Part of this has already been done at the time of writing.
The lens actuator needs to be enabled as well. This has also been done. Auto-focus has not been implemented yet, however.
Some color calibration data must be collected to post-process the raw data coming out of the sensor.

For now, mostly unprocessed images can be captured using Megapixels.

Unlocking the Qualcomm Snapdragon Sensor Core (Part 2)

Yassine Oudjana — Fri, 08 Apr 2022 00:00:00 +0000

Last time we got a step closer to working sensors on mainline Linux by powering on and booting the SLPI successfully. With that easy part out of the way, we can now deal with the hard part: Finding out how to communicate with it and getting something meaningful out of it. Usually when bringing up hardware in the mainline kernel, the downstream kernel drivers, and/or if available, datasheets are used to understand how the hardware works and how to operate it. With it having no open-source drivers nor public documentation however, figuring out how the SSC works will be a challenge on a whole new level.

Without any direct reference material, it would seem to be an impossible task at first, but when looking at it from a different perspective, even the proprietary HAL driver can be used as reference. After all, we know for a fact that it works, so there must be something that can be learned from it.

There are a few ways to understand how the HAL driver works without having its source code on hand. It is possible to disassemble the binary and try to read through assembly code, but getting any meaningful information that way will be quite cumbersome, not to mention the legal uncertainties surrounding this method. A much better alternative in this case would be to run the driver normally, then monitor its interactions with the SSC. After understanding what it does and what for, It should be possible to then imitate it and get identical results from the SSC.

Since it has to talk to hardware, it will have to interact with the kernel using syscalls. Tracing the syscalls it makes will be a good starting point, and for that, the Linux syscall tracer – usually known as strace – is the perfect tool. Using strace will allow for monitoring all interactions between the HAL driver and the Linux kernel, and by extension, the SLPI. It can also attach to already running processes, making it easy to collect some data without any preparation, as long as root privileges are available.

Watching the SSC

The main part of the HAL driver is the sensor service android.hardware.sensors@1.0-service, which is started at boot by the init system. Before attaching strace to it, its PID has to be found:

# pidof android.hardware.sensors@1.0-service
582

Then the -p argument can be used to attach strace to it:

# strace -p 582
strace: Process 582 attached
ioctl(3, BINDER_WRITE_READ, 0x7fc7f43d20) = 0

Quite underwhelming. Could this be the wrong process? Perhaps the right one can be found using htop:

PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
system     20   0 11.4G  2732  2076 S  0.0  0.1  0:01.05 /vendor/bin/hw/android.hardware.sensors@1.0-service
system     20   0 11.4G  2732  2076 S  0.0  0.1  0:00.76 /vendor/bin/hw/android.hardware.sensors@1.0-service
system     20   0 11.4G  2732  2076 S  0.0  0.1  0:00.76 /vendor/bin/hw/android.hardware.sensors@1.0-service
system     20   0 11.4G  2732  2076 S  0.0  0.1  0:35.28 /vendor/bin/hw/android.hardware.sensors@1.0-service
system     20   0 11.4G  2732  2076 S  0.0  0.1  0:00.00 /vendor/bin/hw/android.hardware.sensors@1.0-service
system     20   0 11.4G  2732  2076 S  0.0  0.1  0:00.70 /vendor/bin/hw/android.hardware.sensors@1.0-service
system     20   0 11.4G  2732  2076 S  0.0  0.1  0:00.19 /vendor/bin/hw/android.hardware.sensors@1.0-service
system     20   0 11.4G  2732  2076 S  0.0  0.1  0:00.00 /vendor/bin/hw/android.hardware.sensors@1.0-service
system     20   0 11.4G  2732  2076 S  0.0  0.1  0:00.67 /vendor/bin/hw/android.hardware.sensors@1.0-service
system     20   0 11.4G  2732  2076 S  0.0  0.1  0:00.00 /vendor/bin/hw/android.hardware.sensors@1.0-service
system     20   0 11.4G  2732  2076 S  0.0  0.1  0:00.66 /vendor/bin/hw/android.hardware.sensors@1.0-service
system     20   0 11.4G  2732  2076 S  0.0  0.1  0:00.00 /vendor/bin/hw/android.hardware.sensors@1.0-service
...

There are actually many android.hardware.sensors@1.0-service processes, so it appears that the sensor service has multiple threads running. Fortunately, strace has a -f argument to follow forks, allowing us to attach to all running threads of the sensor service at the same time:

# strace -p 582 -f
strace: Process 582 attached with 70 threads
[pid  2475] ppoll([{fd=107, events=POLLIN}, {fd=106, events=POLLIN}], 2, NULL, NULL, 0 <unfinished ...>
[pid  2473] ppoll([{fd=104, events=POLLIN}, {fd=103, events=POLLIN}], 2, NULL, NULL, 0 <unfinished ...>
[pid  1735] ppoll([{fd=101, events=POLLIN}, {fd=100, events=POLLIN}], 2, NULL, NULL, 0 <unfinished ...>
[pid  1734] ppoll([{fd=15, events=POLLIN}, {fd=14, events=POLLIN}], 2, NULL, NULL, 0 <unfinished ...>
[pid   982] futex(0x715777ad00, FUTEX_WAIT_BITSET_PRIVATE, 747816, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid   980] futex(0x71576f55e8, FUTEX_WAIT_BITSET_PRIVATE, 1219316, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid   977] ppoll([{fd=98, events=POLLIN}, {fd=97, events=POLLIN}], 2, NULL, NULL, 0 <unfinished ...>
[pid   973] ppoll([{fd=95, events=POLLIN}, {fd=94, events=POLLIN}], 2, NULL, NULL, 0 <unfinished ...>
[pid   972] futex(0x72078ff150, FUTEX_WAIT_BITSET_PRIVATE, 4294967294, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid   976] rt_sigtimedwait([RTMIN],  <unfinished ...>
[pid   975] futex(0x72078ff610, FUTEX_WAIT_BITSET_PRIVATE, 4294967294, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid   970] ppoll([{fd=92, events=POLLIN}, {fd=91, events=POLLIN}], 2, NULL, NULL, 0 <unfinished ...>
[pid   969] futex(0x72078ff650, FUTEX_WAIT_BITSET_PRIVATE, 4294967294, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid   966] futex(0x72078fd790, FUTEX_WAIT_BITSET_PRIVATE, 4294967294, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid   967] ppoll([{fd=89, events=POLLIN}, {fd=88, events=POLLIN}], 2, NULL, NULL, 0 <unfinished ...>
[pid   963] ppoll([{fd=86, events=POLLIN}, {fd=85, events=POLLIN}], 2, NULL, NULL, 0 <unfinished ...>
[pid   962] futex(0x72078fd550, FUTEX_WAIT_BITSET_PRIVATE, 4294967294, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid   958] futex(0x72078fdf90, FUTEX_WAIT_BITSET_PRIVATE, 4294967294, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid   959] ppoll([{fd=83, events=POLLIN}, {fd=82, events=POLLIN}], 2, NULL, NULL, 0 <unfinished ...>
[pid   954] ppoll([{fd=80, events=POLLIN}, {fd=79, events=POLLIN}], 2, NULL, NULL, 0 <unfinished ...>
[pid   953] futex(0x72078ff310, FUTEX_WAIT_BITSET_PRIVATE, 4294967294, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid   948] ppoll([{fd=77, events=POLLIN}, {fd=76, events=POLLIN}], 2, NULL, NULL, 0 <unfinished ...>
[pid   947] futex(0x72078ff710, FUTEX_WAIT_BITSET_PRIVATE, 4294967294, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid   939] ppoll([{fd=71, events=POLLIN}, {fd=70, events=POLLIN}], 2, NULL, NULL, 0 <unfinished ...>
[pid   944] ppoll([{fd=74, events=POLLIN}, {fd=73, events=POLLIN}], 2, NULL, NULL, 0 <unfinished ...>
[pid   943] futex(0x72078fec10, FUTEX_WAIT_BITSET_PRIVATE, 4294967294, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid   942] futex(0x72078ff010, FUTEX_WAIT_BITSET_PRIVATE, 4294967294, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid   938] futex(0x72078ff090, FUTEX_WAIT_BITSET_PRIVATE, 4294967294, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid   933] ppoll([{fd=68, events=POLLIN}, {fd=67, events=POLLIN}], 2, NULL, NULL, 0 <unfinished ...>
[pid   932] futex(0x72078ff1d0, FUTEX_WAIT_BITSET_PRIVATE, 4294967294, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid   927] ppoll([{fd=65, events=POLLIN}, {fd=64, events=POLLIN}], 2, NULL, NULL, 0 <unfinished ...>
[pid   921] ppoll([{fd=62, events=POLLIN}, {fd=61, events=POLLIN}], 2, NULL, NULL, 0 <unfinished ...>
[pid   926] futex(0x72078fed50, FUTEX_WAIT_BITSET_PRIVATE, 4294967294, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid   920] futex(0x72078fec90, FUTEX_WAIT_BITSET_PRIVATE, 4294967294, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid   916] ppoll([{fd=56, events=POLLIN}, {fd=55, events=POLLIN}], 2, NULL, NULL, 0 <unfinished ...>
[pid   917] futex(0x72078fe150, FUTEX_WAIT_BITSET_PRIVATE, 4294967294, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid   918] ppoll([{fd=59, events=POLLIN}, {fd=58, events=POLLIN}], 2, NULL, NULL, 0 <unfinished ...>
[pid   915] futex(0x72078fe790, FUTEX_WAIT_BITSET_PRIVATE, 4294967294, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid   913] ppoll([{fd=53, events=POLLIN}, {fd=52, events=POLLIN}], 2, NULL, NULL, 0 <unfinished ...>
[pid   912] futex(0x72078fe350, FUTEX_WAIT_BITSET_PRIVATE, 4294967294, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid   903] ppoll([{fd=50, events=POLLIN}, {fd=49, events=POLLIN}], 2, NULL, NULL, 0 <unfinished ...>
[pid   902] futex(0x72078fe010, FUTEX_WAIT_BITSET_PRIVATE, 4294967294, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid   895] ppoll([{fd=47, events=POLLIN}, {fd=46, events=POLLIN}], 2, NULL, NULL, 0 <unfinished ...>
[pid   894] futex(0x72078fe8d0, FUTEX_WAIT_BITSET_PRIVATE, 4294967294, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid   885] ppoll([{fd=44, events=POLLIN}, {fd=43, events=POLLIN}], 2, NULL, NULL, 0 <unfinished ...>
[pid   884] futex(0x72078febd0, FUTEX_WAIT_BITSET_PRIVATE, 4294967294, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid   876] ppoll([{fd=41, events=POLLIN}, {fd=40, events=POLLIN}], 2, NULL, NULL, 0 <unfinished ...>
[pid   875] futex(0x72078fea90, FUTEX_WAIT_BITSET_PRIVATE, 4294967294, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid   854] ppoll([{fd=35, events=POLLIN}, {fd=34, events=POLLIN}], 2, NULL, NULL, 0 <unfinished ...>
[pid   857] futex(0x72078feb90, FUTEX_WAIT_BITSET_PRIVATE, 4294967294, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid   853] futex(0x72078fdfd0, FUTEX_WAIT_BITSET_PRIVATE, 4294967294, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid   858] ppoll([{fd=38, events=POLLIN}, {fd=37, events=POLLIN}], 2, NULL, NULL, 0 <unfinished ...>
[pid   846] ppoll([{fd=32, events=POLLIN}, {fd=31, events=POLLIN}], 2, NULL, NULL, 0 <unfinished ...>
[pid   845] futex(0x72078fe410, FUTEX_WAIT_BITSET_PRIVATE, 4294967294, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid   832] futex(0x72078fe290, FUTEX_WAIT_BITSET_PRIVATE, 4294967294, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid   828] ppoll([{fd=26, events=POLLIN}, {fd=25, events=POLLIN}], 2, NULL, NULL, 0 <unfinished ...>
[pid   833] ppoll([{fd=29, events=POLLIN}, {fd=28, events=POLLIN}], 2, NULL, NULL, 0 <unfinished ...>
[pid   827] futex(0x72078fe250, FUTEX_WAIT_BITSET_PRIVATE, 4294967294, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid   821] ppoll([{fd=23, events=POLLIN}, {fd=22, events=POLLIN}], 2, NULL, NULL, 0 <unfinished ...>
[pid   820] futex(0x72078fe0d0, FUTEX_WAIT_BITSET_PRIVATE, 4294967294, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid   819] ppoll([{fd=20, events=POLLIN}, {fd=19, events=POLLIN}], 2, NULL, NULL, 0 <unfinished ...>
[pid   780] futex(0x72078fd410, FUTEX_WAIT_BITSET_PRIVATE, 4294967294, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid   774] ppoll([{fd=12, events=POLLIN}, {fd=11, events=POLLIN}], 2, NULL, NULL, 0 <unfinished ...>
[pid   773] futex(0x72078fd8d0, FUTEX_WAIT_BITSET_PRIVATE, 4294967294, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid   771] futex(0x72078fd4d0, FUTEX_WAIT_BITSET_PRIVATE, 4294967294, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid   613] ppoll([{fd=8, events=POLLIN}, {fd=7, events=POLLIN}], 2, NULL, NULL, 0 <unfinished ...>
[pid   612] read(6,  <unfinished ...>
[pid   610] pselect6(6, [5], NULL, [5], NULL, NULL <unfinished ...>
[pid   582] ioctl(3, BINDER_WRITE_READ <unfinished ...>
[pid   818] futex(0x72078fe210, FUTEX_WAIT_BITSET_PRIVATE, 4294967294, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid  1734] <... ppoll resumed> )       = 1 ([{fd=14, revents=POLLIN}])
[pid  1734] recvfrom(14, "\4\226\5\3\0$\0\1\4\0\275\23zo\2\10\0\331\0\200R\203`\336\26\3\4\0\0\0\0\0"..., 56, MSG_DONTWAIT, {sa_family=AF_IB, sa_data="\0\0\2\0\0\0\1\0\0\0)\0\0\0\0\0\0\0"}, [20]) = 43
[pid  1734] rt_sigprocmask(0x539a1968 /* SIG_??? */, NULL, [USR2 RTMIN], 8) = 0
[pid  1734] rt_sigprocmask(0x539a1958 /* SIG_??? */, NULL, [USR2 RTMIN], 8) = 0
[pid  1734] futex(0x72078fd410, FUTEX_WAKE_PRIVATE, 2147483647) = 1
[pid   780] <... futex resumed> )       = 0
[pid  1734] ppoll([{fd=15, events=POLLIN}, {fd=14, events=POLLIN}], 2, NULL, NULL, 0 <unfinished ...>
[pid   780] futex(0x72078fd410, FUTEX_WAIT_BITSET_PRIVATE, 4294967294, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid  1734] <... ppoll resumed> )       = 1 ([{fd=14, revents=POLLIN}])
[pid  1734] recvfrom(14, "\4\227\5\3\0$\0\1\4\0\23\24\177o\2\10\0\263\217\263\246\205`\336\26\3\4\0\0\0\0\0"..., 56, MSG_DONTWAIT, {sa_family=AF_IB, sa_data="\0\0\2\0\0\0\1\0\0\0)\0\0\0\0\0\0\0"}, [20]) = 43
[pid  1734] rt_sigprocmask(0x539a1968 /* SIG_??? */, NULL, [USR2 RTMIN], 8) = 0
[pid  1734] rt_sigprocmask(0x539a1958 /* SIG_??? */, NULL, [USR2 RTMIN], 8) = 0
[pid  1734] futex(0x72078fd410, FUTEX_WAKE_PRIVATE, 2147483647) = 1
[pid  1734] ppoll([{fd=15, events=POLLIN}, {fd=14, events=POLLIN}], 2, NULL, NULL, 0 <unfinished ...>
[pid   780] <... futex resumed> )       = 0
[pid   780] futex(0x72078fd410, FUTEX_WAIT_BITSET_PRIVATE, 4294967294, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid  1734] <... ppoll resumed> )       = 1 ([{fd=14, revents=POLLIN}])
[pid  1734] recvfrom(14, "\4\230\5\3\0$\0\1\4\0o\24\204o\2\10\0\22\216\352\372\207`\336\26\3\4\0\0\0\0\0"..., 56, MSG_DONTWAIT, {sa_family=AF_IB, sa_data="\0\0\2\0\0\0\1\0\0\0)\0\0\0\0\0\0\0"}, [20]) = 43
[pid  1734] rt_sigprocmask(0x539a1968 /* SIG_??? */, NULL, [USR2 RTMIN], 8) = 0
[pid  1734] rt_sigprocmask(0x539a1958 /* SIG_??? */, NULL, [USR2 RTMIN], 8) = 0
[pid  1734] futex(0x72078fd410, FUTEX_WAKE_PRIVATE, 2147483647) = 1
[pid   780] <... futex resumed> )       = 0
[pid  1734] ppoll([{fd=15, events=POLLIN}, {fd=14, events=POLLIN}], 2, NULL, NULL, 0 <unfinished ...>
[pid   780] futex(0x72078fd410, FUTEX_WAIT_BITSET_PRIVATE, 4294967294, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid  1734] <... ppoll resumed> )       = 1 ([{fd=14, revents=POLLIN}])
[pid  1734] recvfrom(14, "\4\231\5\3\0$\0\1\4\0\374\24\211o\2\10\0 \3357O\212`\336\26\3\4\0\0\0\0\0"..., 56, MSG_DONTWAIT, {sa_family=AF_IB, sa_data="\0\0\2\0\0\0\1\0\0\0)\0\0\0\0\0\0\0"}, [20]) = 43
[pid  1734] rt_sigprocmask(0x539a1968 /* SIG_??? */, NULL, [USR2 RTMIN], 8) = 0
[pid  1734] rt_sigprocmask(0x539a1958 /* SIG_??? */, NULL, [USR2 RTMIN], 8) = 0
[pid  1734] futex(0x72078fd410, FUTEX_WAKE_PRIVATE, 2147483647) = 1
[pid  1734] ppoll([{fd=15, events=POLLIN}, {fd=14, events=POLLIN}], 2, NULL, NULL, 0 <unfinished ...>
[pid   780] <... futex resumed> )       = 0
[pid   780] futex(0x72078fd410, FUTEX_WAIT_BITSET_PRIVATE, 4294967294, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid  1734] <... ppoll resumed> )       = 1 ([{fd=14, revents=POLLIN}])
[pid  1734] recvfrom(14, "\4\235\5\3\0$\0\1\4\0-\26\235o\2\10\0\5\224\365\237\223`\336\26\3\4\0\0\0\0\0"..., 56, MSG_DONTWAIT, {sa_family=AF_IB, sa_data="\0\0\2\0\0\0\1\0\0\0)\0\0\0\0\0\0\0"}, [20]) = 43
[pid  1734] rt_sigprocmask(0x539a1968 /* SIG_??? */, NULL, [USR2 RTMIN], 8) = 0
[pid  1734] rt_sigprocmask(0x539a1958 /* SIG_??? */, NULL, [USR2 RTMIN], 8) = 0
[pid  1734] futex(0x72078fd410, FUTEX_WAKE_PRIVATE, 2147483647) = 1
[pid  1734] ppoll([{fd=15, events=POLLIN}, {fd=14, events=POLLIN}], 2, NULL, NULL, 0 <unfinished ...>
[pid   780] <... futex resumed> )       = 0
[pid   780] futex(0x72078fd410, FUTEX_WAIT_BITSET_PRIVATE, 4294967294, NULL, FUTEX_BITSET_MATCH_ANY

Now that is more like it. There is a lot going on, so filtering the syscalls would be a good idea. Since we are looking for data transferred between SSC and the sensor service, we can focus on sendto and recvfrom calls. This can be done using the -e argument. The -xx argument is also added to print all strings in hexadecimal format, and -s 1024 to increase the string size limit so that no characters get truncated:

# strace -p 582 -f -e trace=sendto,recvfrom -xx -s 1024
strace: Process 582 attached with 70 threads

Nothing?

This is being done with the screen off, so the silence might be due to the sensors being suspended to save power. One sensor should be active however; as the phone can be woken up by waving a hand in front of the proximity sensor. To confirm, the proximity sensor is covered then uncovered a few times:

[pid   977] recvfrom(97, "\x04\xad\x00\x05\x00\x1a\x00\x01\x01\x00\x1a\x02\x04\x00\xf2\xfe\xf4\x71\x03\x0c\x00\x00\x00\x01\x00\xa0\x3a\x29\xb2\x00\x00\x00\x00", 820, MSG_DONTWAIT, {sa_family=AF_IB, sa_data="\x00\x00\x02\x00\x00\x00\x09\x00\x00\x00\x3f\x00\x00\x00\x00\x00\x00\x00"}, [20]) = 33
[pid   977] recvfrom(97, "\x04\xae\x00\x05\x00\x1a\x00\x01\x01\x00\x1a\x02\x04\x00\x09\x26\xf5\x71\x03\x0c\x00\x00\x00\x00\x00\xa0\x3a\x29\xb2\x00\x00\x00\x00", 820, MSG_DONTWAIT, {sa_family=AF_IB, sa_data="\x00\x00\x02\x00\x00\x00\x09\x00\x00\x00\x3f\x00\x00\x00\x00\x00\x00\x00"}, [20]) = 33
[pid   582] sendto(14, "\x00\xc0\x01\x02\x00\x04\x00\x10\x01\x00\x01", 11, MSG_DONTWAIT, {sa_family=AF_IB, sa_data="\x00\x00\x02\x00\x00\x00\x01\x00\x00\x00\x29\x00\x00\x00\xd8\x33\xf4\xc7"}, 20) = 11
[pid  1734] recvfrom(14, "\x02\xc0\x01\x02\x00\x29\x00\x02\x02\x00\x00\x00\x10\x04\x00\xe0\x2d\xf5\x71\x11\x08\x00\xfb\xcb\x93\x10\xab\x61\xde\x16\x13\x04\x00\x00\x00\x00\x00\x14\x08\x00\xa9\x13\x2e\xdc\x0e\x35\x00\x00", 56, MSG_DONTWAIT, {sa_family=AF_IB, sa_data="\x00\x00\x02\x00\x00\x00\x01\x00\x00\x00\x29\x00\x00\x00\x00\x00\x00\x00"}, [20]) = 48
[pid   582] sendto(94, "\x00\x7e\x00\x02\x00\x26\x00\x01\x01\x00\x28\x02\x01\x00\x00\x03\x04\x00\x00\x00\x05\x00\x04\x0c\x00\xff\xff\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x11\x05\x00\x00\x00\x00\x00\x01", 45, MSG_DONTWAIT, {sa_family=AF_IB, sa_data="\x00\x00\x02\x00\x00\x00\x09\x00\x00\x00\x3f\x00\x00\x00\x58\x34\xf4\xc7"}, 20) = 45
[pid   973] recvfrom(94, "\x04\x3f\x00\x05\x00\x1a\x00\x01\x01\x00\x1f\x02\x04\x00\x09\x26\xf5\x71\x03\x0c\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00", 820, MSG_DONTWAIT, {sa_family=AF_IB, sa_data="\x00\x00\x02\x00\x00\x00\x09\x00\x00\x00\x3f\x00\x00\x00\x00\x00\x00\x00"}, [20]) = 33
[pid   973] recvfrom(94, "\x02\x7e\x00\x02\x00\x09\x00\x02\x02\x00\x00\x00\x10\x01\x00\x1f", 820, MSG_DONTWAIT, {sa_family=AF_IB, sa_data="\x00\x00\x02\x00\x00\x00\x09\x00\x00\x00\x3f\x00\x00\x00\x00\x00\x00\x00"}, [20]) = 16
[pid  1734] recvfrom(14, "\x04\x15\x06\x03\x00\x24\x00\x01\x04\x00\x62\x3a\xf5\x71\x02\x08\x00\xa6\xa0\x66\x16\xab\x61\xde\x16\x03\x04\x00\x00\x00\x00\x00\x10\x08\x00\x0a\xe3\x00\xe2\x0e\x35\x00\x00", 56, MSG_DONTWAIT, {sa_family=AF_IB, sa_data="\x00\x00\x02\x00\x00\x00\x01\x00\x00\x00\x29\x00\x00\x00\x00\x00\x00\x00"}, [20]) = 43
[pid   982] sendto(94, "\x00\x7f\x00\x00\x00\x00\x00", 7, MSG_DONTWAIT, {sa_family=AF_IB, sa_data="\x00\x00\x02\x00\x00\x00\x09\x00\x00\x00\x3f\x00\x00\x00\xb8\xf0\xa9\x53"}, 20) = 7
[pid   973] recvfrom(94, "\x02\x7f\x00\x00\x00\x05\x00\x02\x02\x00\x00\x00", 820, MSG_DONTWAIT, {sa_family=AF_IB, sa_data="\x00\x00\x02\x00\x00\x00\x09\x00\x00\x00\x3f\x00\x00\x00\x00\x00\x00\x00"}, [20]) = 12
[pid   977] recvfrom(97, "\x04\xaf\x00\x05\x00\x1a\x00\x01\x01\x00\x1a\x02\x04\x00\x69\x8f\xf5\x71\x03\x0c\x00\x00\x00\x01\x00\xa0\x3a\x29\xb2\x00\x00\x00\x00", 820, MSG_DONTWAIT, {sa_family=AF_IB, sa_data="\x00\x00\x02\x00\x00\x00\x09\x00\x00\x00\x3f\x00\x00\x00\x00\x00\x00\x00"}, [20]) = 33
[pid   977] recvfrom(97, "\x04\xb0\x00\x05\x00\x1a\x00\x01\x01\x00\x1a\x02\x04\x00\xce\xa3\xf5\x71\x03\x0c\x00\x00\x00\x00\x00\xa0\x3a\x29\xb2\x00\x00\x00\x00", 820, MSG_DONTWAIT, {sa_family=AF_IB, sa_data="\x00\x00\x02\x00\x00\x00\x09\x00\x00\x00\x3f\x00\x00\x00\x00\x00\x00\x00"}, [20]) = 33
[pid   977] recvfrom(97, "\x04\xb1\x00\x05\x00\x1a\x00\x01\x01\x00\x1a\x02\x04\x00\x94\x10\xf6\x71\x03\x0c\x00\x00\x00\x01\x00\xa0\x3a\x29\xb2\x00\x00\x00\x00", 820, MSG_DONTWAIT, {sa_family=AF_IB, sa_data="\x00\x00\x02\x00\x00\x00\x09\x00\x00\x00\x3f\x00\x00\x00\x00\x00\x00\x00"}, [20]) = 33
[pid   977] recvfrom(97, "\x04\xb2\x00\x05\x00\x1a\x00\x01\x01\x00\x1a\x02\x04\x00\x13\x19\xf6\x71\x03\x0c\x00\x00\x00\x00\x00\xa0\x3a\x29\xb2\x00\x00\x00\x00", 820, MSG_DONTWAIT, {sa_family=AF_IB, sa_data="\x00\x00\x02\x00\x00\x00\x09\x00\x00\x00\x3f\x00\x00\x00\x00\x00\x00\x00"}, [20]) = 33
[pid   977] recvfrom(97, "\x04\xb3\x00\x05\x00\x1a\x00\x01\x01\x00\x1a\x02\x04\x00\xdf\x1f\xf6\x71\x03\x0c\x00\x00\x00\x01\x00\xa0\x3a\x29\xb2\x00\x00\x00\x00", 820, MSG_DONTWAIT, {sa_family=AF_IB, sa_data="\x00\x00\x02\x00\x00\x00\x09\x00\x00\x00\x3f\x00\x00\x00\x00\x00\x00\x00"}, [20]) = 33
[pid   977] recvfrom(97, "\x04\xb4\x00\x05\x00\x1a\x00\x01\x01\x00\x1a\x02\x04\x00\x5f\x28\xf6\x71\x03\x0c\x00\x00\x00\x00\x00\xa0\x3a\x29\xb2\x00\x00\x00\x00", 820, MSG_DONTWAIT, {sa_family=AF_IB, sa_data="\x00\x00\x02\x00\x00\x00\x09\x00\x00\x00\x3f\x00\x00\x00\x00\x00\x00\x00"}, [20]) = 33

And sure enough, data starts coming in. A recvfrom call is traced each time the proximity sensor is covered or uncovered. There are a few extra sendto and recvfrom calls after the first time the sensor is covered, so it seems that something happens when the screen turns on. That will not be important now though.

Reading the data

Right now all we have is a bunch of strace output lines with traced syscalls and the arguments passed through them. Since we are after raw data, some work has to be done on those lines. To start analyzing the data format, a single recvfrom is taken:

[pid   977] recvfrom(97, "\x04\xb1\x00\x05\x00\x1a\x00\x01\x01\x00\x1a\x02\x04\x00\x94\x10\xf6\x71\x03\x0c\x00\x00\x00\x01\x00\xa0\x3a\x29\xb2\x00\x00\x00\x00", 820, MSG_DONTWAIT, {sa_family=AF_IB, sa_data="\x00\x00\x02\x00\x00\x00\x09\x00\x00\x00\x3f\x00\x00\x00\x00\x00\x00\x00"}, [20]) = 33

Then the data, which is the second argument of recvfrom, is extracted and formatted nicely:

# printf "\x04\xb1\x00\x05\x00\x1a\x00\x01\x01\x00\x1a\x02\x04\x00\x94\x10\xf6\x71\x03\x0c\x00\x00\x00\x01\x00\xa0\x3a\x29\xb2\x00\x00\x00\x00" | xxd -g 1
00000000: 04 b1 00 05 00 1a 00 01 01 00 1a 02 04 00 94 10  ................
00000010: f6 71 03 0c 00 00 00 01 00 a0 3a 29 b2 00 00 00  .q........:)....
00000020: 00                                               .

These are the bytes received, put in a single line:

04 b1 00 05 00 1a 00 01 01 00 1a 02 04 00 94 10 f6 71 03 0c 00 00 00 01 00 a0 3a 29 b2 00 00 00 00

There is a prominent pattern that can be quickly identified here: There are several non-zero bytes followed by zeroes, such as b1 00, 05 00, 1a 00, and 0c 00 00 00. This is a strong indicator that this data is little-endian, meaning that integers that are 2 bytes or longer are represented in such a way that their least significant byte comes first, followed by the second least significant byte, and so on.

To interpret the data, it might help to start off with some guesses on the fields it is composed of. At a first glance, it looks like it can be split into these fields, resembled with a C struct:

struct prox_pkt {
	u8 var1;	// 04
	u16 var2;	// b1 00
	u16 var3;	// 05 00
	u16 var4;	// 1a 00
	u8 var5;	// 01
	u16 var6;	// 01 00
	u8 var7;	// 1a
	u8 var8;	// 02
	u16 var9;	// 04 00
	u8 var10;	// 94
	u8 var11;	// 10
	u8 var12;	// f6
	u8 var13;	// 71
	u8 var14;	// 03
	u32 var15;	// 0c 00 00 00
	u16 var16;	// 01 00
	u8 var17;	// a0
	u8 var18;	// 3a
	u8 var19;	// 29
	u8 var20;	// b2
	u32 var21;	// 00 00 00 00
};

struct prox_pkt pkt1 = {0x04, 0xb1, 0x05, 0x1a, 0x01, 0x01, 0x1a, 0x02, 0x04, 0x94, 0x10, 0xf6, 0x71, 0x03, 0x0c, 0x01, 0xa0, 0x3a, 0x29, 0xb2, 0x00};

Again, the little-endianness of the data is clearly seen in every u16 and u32 field, which are 2 bytes and 4 bytes long respectively. The zeroes helped with identifying the sizes of those fields. There might be additional 2 or 4 byte long fields that are holding large integers where their most significant bytes are non-zero, making them difficult to identify at this stage.

Next, the data is analyzed further by comparing 2 consecutive lines. The steps above are repeated with the next line:

[pid   977] recvfrom(97, "\x04\xb2\x00\x05\x00\x1a\x00\x01\x01\x00\x1a\x02\x04\x00\x13\x19\xf6\x71\x03\x0c\x00\x00\x00\x00\x00\xa0\x3a\x29\xb2\x00\x00\x00\x00", 820, MSG_DONTWAIT, {sa_family=AF_IB, sa_data="\x00\x00\x02\x00\x00\x00\x09\x00\x00\x00\x3f\x00\x00\x00\x00\x00\x00\x00"}, [20]) = 33

04 b2 00 05 00 1a 00 01 01 00 1a 02 04 00 13 19 f6 71 03 0c 00 00 00 00 00 a0 3a 29 b2 00 00 00 00

The format seems identical, so it should be possible to compare this with the previous line directly.

struct prox_pkt pkt1 = {0x04, 0xb1, 0x05, 0x1a, 0x01, 0x01, 0x1a, 0x02, 0x04, 0x94, 0x10, 0xf6, 0x71, 0x03, 0x0c, 0x01, 0xa0, 0x3a, 0x29, 0xb2, 0x00};
struct prox_pkt pkt2 = {0x04, 0xb2, 0x05, 0x1a, 0x01, 0x01, 0x1a, 0x02, 0x04, 0x13, 0x19, 0xf6, 0x71, 0x03, 0x0c, 0x00, 0xa0, 0x3a, 0x29, 0xb2, 0x00};

Most of the values remained constant. The only values that changed are var2 which increased by 1, var10 and var11 which changed somewhat significantly, and var16 which changed from 1 to 0.

Looking at a larger set of lines should make it possible to identify patterns and find constant values with more certainty, as well as find some of the hiding large integers mentioned earlier.

Formatting each line from the strace output manually will take a lot of time. With some quick Python magic though:

import sys
import re

i = open(sys.argv[1], "r")

ilines = i.readlines()

for iline in ilines:
	if "ENOMSG" in iline:
		continue
	if "sendto" not in iline and "recvfrom" not in iline:
		continue

	line = ""
	line += re.findall("\d+", iline)[0] + "\t"

	if "sendto" in iline:
		line += "send"
	elif "recvfrom" in iline:
		line += "recv"
	line += "\t"

	fd = re.findall("\(\d\d", iline)
	if len(fd) == 0:
		continue

	line += fd[0][1::] + "\t"

	rawdata = re.findall("[^\"][x][0-9a-f][0-9a-f][^\"]+", iline)
	if len(rawdata) == 0:
		continue
	rawdata = rawdata[0][2::].split("\\x")

	data = []

	for rawbyte in rawdata:
		data.append(int(rawbyte, 16))

	line += "|"
	for byte in data:
		line += chr(byte) if byte <= 127 and byte >= 32 else "."
	line += "|\t\t"
	for byte in data:
		line += "{:02x}".format(byte) + " "

	print(line)

i.close()

The strace output can be quickly converted into a nice and easy to read format, including an ASCII representation that should reveal any text possibly carried in the packets:

recv	97	|.................q........:).....|		04 ad 00 05 00 1a 00 01 01 00 1a 02 04 00 f2 fe f4 71 03 0c 00 00 00 01 00 a0 3a 29 b2 00 00 00 00
recv	97	|...............&.q........:).....|		04 ae 00 05 00 1a 00 01 01 00 1a 02 04 00 09 26 f5 71 03 0c 00 00 00 00 00 a0 3a 29 b2 00 00 00 00
send	14	|...........|		00 c0 01 02 00 04 00 10 01 00 01
recv	14	|.....)..........-.q........a.................5..|		02 c0 01 02 00 29 00 02 02 00 00 00 10 04 00 e0 2d f5 71 11 08 00 fb cb 93 10 ab 61 de 16 13 04 00 00 00 00 00 14 08 00 a9 13 2e dc 0e 35 00 00
send	94	|.~...&....(..................................|		00 7e 00 02 00 26 00 01 01 00 28 02 01 00 00 03 04 00 00 00 05 00 04 0c 00 ff ff 00 00 00 00 00 00 00 00 00 00 11 05 00 00 00 00 00 01
recv	94	|.?.............&.q...............|		04 3f 00 05 00 1a 00 01 01 00 1f 02 04 00 09 26 f5 71 03 0c 00 00 00 00 00 00 00 00 00 00 00 00 00
recv	94	|.~..............|		02 7e 00 02 00 09 00 02 02 00 00 00 10 01 00 1f
recv	14	|.....$....b:.q.....f..a.................5..|		04 15 06 03 00 24 00 01 04 00 62 3a f5 71 02 08 00 a6 a0 66 16 ab 61 de 16 03 04 00 00 00 00 00 10 08 00 0a e3 00 e2 0e 35 00 00
send	94	|......|		00 7f 00 00 00 00 00
recv	94	|...........|		02 7f 00 00 00 05 00 02 02 00 00 00
recv	97	|..............i..q........:).....|		04 af 00 05 00 1a 00 01 01 00 1a 02 04 00 69 8f f5 71 03 0c 00 00 00 01 00 a0 3a 29 b2 00 00 00 00
recv	97	|.................q........:).....|		04 b0 00 05 00 1a 00 01 01 00 1a 02 04 00 ce a3 f5 71 03 0c 00 00 00 00 00 a0 3a 29 b2 00 00 00 00
recv	97	|.................q........:).....|		04 b1 00 05 00 1a 00 01 01 00 1a 02 04 00 94 10 f6 71 03 0c 00 00 00 01 00 a0 3a 29 b2 00 00 00 00
recv	97	|.................q........:).....|		04 b2 00 05 00 1a 00 01 01 00 1a 02 04 00 13 19 f6 71 03 0c 00 00 00 00 00 a0 3a 29 b2 00 00 00 00
recv	97	|.................q........:).....|		04 b3 00 05 00 1a 00 01 01 00 1a 02 04 00 df 1f f6 71 03 0c 00 00 00 01 00 a0 3a 29 b2 00 00 00 00
recv	97	|.............._(.q........:).....|		04 b4 00 05 00 1a 00 01 01 00 1a 02 04 00 5f 28 f6 71 03 0c 00 00 00 00 00 a0 3a 29 b2 00 00 00 00

Columns from left to right: PID, direction (send/receive), socket file descriptor, ASCII representation, bytes in hexadecimal.

Looking at the last six packets, some interesting patterns are found:

The second byte in each packet (var2) is incremented by 1 on each packet. It might be some sort of a counter.
Bytes 14, 15 and 16 seem to be part of a single variable that is always increasing. Along with byte 17, they likely make up an unsigned 32-bit timestamp, as knowing the time when a sensor reading was taken is needed for calculating relative changes among other things, and there are no other bytes that might represent time here.
The 24^th Byte alternates between 1 and 0. This makes it almost certainly a boolean carrying the proximity sensor data, as it exactly matches the covering/uncovering of the sensor.

Another pattern can be found by taking a look at all the packets including the ones in the first sendto/recvfrom syscalls:

The first byte is always 0 when a packet is sent, and either 2 or 4 when it is received. This might be a header with some specific meaning for each of those three values.

The other values remain constant, so it will not be possible to understand their purpose without some external reference or context that would give them meaning.

With the new findings, an updated struct prox_pkt would look like this:

struct prox_pkt {
	u8 header;
	u16 counter;
	u16 var3;
	u16 var4;
	u8 var5;
	u16 var6;
	u8 var7;
	u8 var8;
	u16 var9;
	u32 timestamp;
	u8 var11;
	u32 var12;
	bool sensor_val;
	u8 var14;
	u8 var15;
	u8 var16;
	u8 var17;
	u8 var18;
	u32 var19;
};

Some important fields have been identified so far, but the purpose of most still remains unknown.

The next step would be to identify the format of this packet, and for that, there are a couple of clues. First of all, the packets traced are moving to and from a Hexagon DSP on a Qualcomm SoC. These tend to use a certain protocol, whose packets – or messages – all have a unique header section. One that has, like in the beginning of our struct prox_pkt, a u8 followed by three u16s…

Introducing QMI

The Qualcomm MSM Interface is a protocol originally developed for interfacing with baseband processors on the MSM line of SoCs, that eventually ended up as the primary interface between the application processor and remote processors on Qualcomm SoCs such as MPSS, ADSP, and now SLPI.

QMI messages

Processors on the SoC communicate by sending messages over QMI, which are transferred through shared memory. As mentioned earlier, QMI messages have a common header:

/**
 * struct qmi_header - wireformat header of QMI messages
 * @type:	type of message
 * @txn_id:	transaction id
 * @msg_id:	message id
 * @msg_len:	length of message payload following header
 */
struct qmi_header {
	u8 type;
	u16 txn_id;
	u16 msg_id;
	u16 msg_len;
} __packed;

To understand what these fields mean, it would help to understand a couple of things about QMI first:

QMI messages are classified into three types:
1. Request, which is sent from a processor to another, which is expected to send back a message of the second type:
2. Response, Sent after receiving a request.
3. Indication, which is sent without expecting a response.
A pair of request and response or a single indication is called a transmission.

Going back to the QMI header:

type can be one of three possible values, which match the three QMI message types:
```
#define QMI_REQUEST	0
#define QMI_RESPONSE	2
#define QMI_INDICATION	4
```
txn_id is a unique identifier for a transmission. It is generally incremented after each transmission, causing the counting pattern observed in the proximity sensor messages. This identifier allows for pairing requests with their responses; as matching pairs have the same identifier.
msg_id is used to identify a specific type of message. Messages of a certain type have the same format and purpose. This property will be explored later.
msg_len is the length of the message in bytes, header excluded.

Adjusting for the QMI header, struct prox_pkt will look like this:

struct prox_pkt {
	struct qmi_header;
	u8 var5
	u16 var6;
	u8 var7;
	u8 var8;
	u16 var9;
	u32 timestamp;
	u8 var11;
	u32 var12;
	bool sensor_val;
	u8 var14;
	u8 var15;
	u8 var16;
	u8 var17;
	u8 var18;
	u32 var19;
};

Reading a QMI header

Taking the first proximity sensor message from before:

04 b2 00 05 00 1a 00 01 01 00 1a 02 04 00 13 19 f6 71 03 0c 00 00 00 00 00 a0 3a 29 b2 00 00 00 00

The header, which is the first seven bytes, can be extracted:

04 b2 00 05 00 1a 00

To read the header, the bytes are put in a struct qmi_header. The first byte is u8 type, and the following three pairs of bytes make up u16 txn_id, msg_id, msg_len:

struct qmi_header prox_msg_hdr {
	type = QMI_INDICATION,
	txn_id = 0xb2,
	msg_id = 0x05,
	msg_len = 0x1a,
};

This message is SSC indicating the state of the proximity sensor, so its type being QMI_INDICATION makes sense as it would not need a response. txn_id does not carry any significant meaning; as it is only used to track the message as it moves around, and since this message is an indication, there is no request to pair it to. msg_id specifies the type of this message and identifies its format so that it can be decoded properly. This property will become useful at a later stage. Finally, msg_len tells us that the following message should be 0x1a, or 26 bytes long.

QMI encoding scheme

QMI messages are encoded in a Type–Length–Value (TLV for short) scheme, where data fields, which are usually called elements in QMI, are put in blocks, each having a type value that is used to identify the field, a length value specifying the length of the data carried in the field, and a value which holds the actual data.

Decoding QMI messages

Since QMI messages are TLV-encoded, many of the unknown variables in the struct prox_pkt guessed earlier are in fact types and lengths rather than actual values. Now that the encoding scheme is known, it should be possible to decode the data captured in the previous stage. The message elements can be discovered by going through the data byte by byte and splitting them into TLV blocks.

Looking at the QMI encoding/decoding helper in Linux, it can be understood how the TLV blocks are encoded. The first thing in the file is a macro used to encode a TLV block:

#define QMI_ENCDEC_ENCODE_TLV(type, length, p_dst) do { \
	*p_dst++ = type; \
	*p_dst++ = ((u8)((length) & 0xFF)); \
	*p_dst++ = ((u8)(((length) >> 8) & 0xFF)); \
} while (0)

This writes one byte for type into a buffer, followed by two bytes for length. These sizes are further confirmed after looking a bit deeper into the file:

#define TLV_LEN_SIZE sizeof(u16)
#define TLV_TYPE_SIZE sizeof(u8)

A message is now taken from before

04 b1 00 05 00 1a 00 01 01 00 1a 02 04 00 94 10 f6 71 03 0c 00 00 00 01 00 a0 3a 29 b2 00 00 00 00

and is split into TLV blocks by taking one byte for type, two bytes for length, then length bytes for the actual value. This is repeated until all blocks are found:

b1 00 05 00 1a 00

01 00 1a
04 00 94 10 f6 71
0c 00 00 00 01 00 a0 3a 29 b2 00 00 00 00

The first line is the header which was discussed earlier. The following lines are the TLV blocks of elements carried in the message, with the first byte being the type, the next 2 bytes the length, and the remaining bytes the value. A closer look is taken at each element:

The first element is a single byte, which makes it likely to be a u8. Its value is 0x1a, which remains constant across all messages traced before. Being the first element in the message, it might be some sort of sensor identifier. This can be confirmed later by capturing messages carrying data from other sensors, then comparing this value to the one in those messages.
The second element is 4 bytes long. This is the timestamp from before, which is now confirmed to be a u32.
The third element is 12 bytes long. It might be a u8 array with a size of 12, but it could also be a u16 array with a size of 6, or a u32 array with a size of 3. Part of this element’s value is the proximity sensor value sensor_val that was discovered earlier. Considering that this message carries sensor data, and many sensors have 3 axes, the latter possibility is more likely even if the proximity sensor only outputs a boolean; as the message format might be the same for several, if not all sensor types.

Now to put these elements in a new struct:

struct prox_ind {
	u8 sensor_id;
	u32 timestamp;
	u32 data[3];
};

That looks much better than the previous struct prox_pkt.

Now to create a struct prox_ind with the data from the message above:

struct prox_ind ind1 = {
	.sensor_id = 0x1a,
	.timestamp = 1911951508,
	.data = {
		1,
		2989046432,
		0
	}
};

The consecutive message can also be decoded for comparison:

04 b2 00 05 00 1a 00 01 01 00 1a 02 04 00 13 19 f6 71 03 0c 00 00 00 00 00 a0 3a 29 b2 00 00 00 00

b2 00 05 00 1a 00

01 00 1a
04 00 13 19 f6 71
0c 00 00 00 00 00 a0 3a 29 b2 00 00 00 00

struct prox_ind ind2 = {
	.sensor_id = 0x1a,
	.timestamp = 1911953683,
	.data = {
		0,
		2989046432,
		0
	}
};

The timestamp increased in the second message, and data[0] changed to 0 indicating that the proximity sensor was uncovered. There is a strange large constant value in data[1] though. It might be good to try to interpret that at some later stage.

Next steps

After developing a method to capture and decode messages transferred between the android sensor service and the SSC, more information can be collected on their operation by analyzing messages at different times, such as with the screen on or with an app using data from specific sensors. However, all of these discoveries would be useless without figuring out how the sensor service sends and receives these messages, and how to do the same thing on mainline Linux. This, along with a second method to capture messages, will be explored next time.

Fingerprint sensors on tuxified Android phones: Impossible? (Part 1)

Yassine Oudjana — Sun, 12 Dec 2021 00:00:00 +0000

Fast authentication is one aspect where Linux phones are still lacking. As of now, the only common way to unlock a phone running Linux is to use a PIN or password. While that may have security benefits to some, it is inconvenient for many, who would rather use a faster method to unlock their phones such as a fingerprint sensor. Possibly the only phone running Linux that can be unlocked with a fingerprint at the moment is a PinePhone fitted with a prototype fingerprint sensor case. Linux phones such as the PinePhone (by default at least) and the Librem 5 might not have fingerprint sensors, but what about Android phones running Linux? Nowadays even the cheapest Android phones have some sort of fingerprint sensor, so why don’t we simply run Linux on those to get fingerprint authentication on a Linux phone? Making an attempt at enabling the fingerprint sensor of an actual Android device on the mainline Linux kernel should provide us with some answers.

The Xiaomi Mi Note 2 will be used as a subject in this experiment. It has a fingerprint sensor built into its home button, and it can already run the mainline Linux kernel to a great extent, so enabling the fingerprint sensor would not take long, or would it?

Enabling the fingerprint sensor

Collecting information

As usual, the first place to look for hardware information is the vendor device tree found in the downstream kernel source. This node seems interesting:

	fpc_fpc1020 {
		compatible = "fpc,fpc1020";
		fpc,irq-gpio	= <&tlmm 121 0x2001>;
		fpc,fp-id-gpio = <&pm8994_gpios 11 0x00>;
		pinctrl-names = "pmx_fp_active", "pmx_fp_suspend";
		pinctrl-0 = <&fp_active>;
		pinctrl-1 = <&fp_suspend>;
		vcc_spi-supply	= <&pm8994_s4>;
		vdd_ana-supply	= <&pm8994_s4>;
		vdd_io-supply	= <&pm8994_s4>;
	};

The official product page from Fingerprint Cards is the first match when searching for fpc1020:

The FPC1020 is a new compact CMOS fingerprint sensor with several significant advantages. It delivers best-in-class image quality with 256 gray-scale values in every single programmable pixel. The sensor with its 3D pixel-sensing technology can read virtually any finger – dry or wet. Thanks to the extremely hard and durable surface coating capable of withstanding more than 10 million finger placements, the FPC1020 is protected against ESD well above 30 kV as well as against scratches and everyday wear and tear.

Looks well enough like a fingerprint sensor.

This line gives us another hint:

		vcc_spi-supply	= <&pm8994_s4>;

It makes clear that this sensor communicates over a Serial Peripheral Interface, or SPI for short. Strangely however, this node is a child of the root node, when it is supposed to be a subnode of the SPI master node the sensor is wired up to. In fact, there is no indication to be found in this device tree of which SPI master this sensor is connected to. This leaves us with the other main source of hardware information: Schematics.

Looking at the SoC GPIO page, the connection between fingerprint sensor and SoC is quickly identified by the FP prefix on the pin names:

The MSM8996 pin controller driver then makes it clear that SPI 5 is exposed on these pins:

static const char * const blsp_spi5_groups[] = {
	"gpio81", "gpio82", "gpio83", "gpio84",
};

Enabling the SPI master

If this works, it should become possible to communicate with the fingerprint sensor.

The SoC DTS is lacking a node for SPI 5, so it is the first thing that should be added. Pin state nodes are added to configure the pin controller, then a node for the SPI master is added:

--- a/arch/arm64/boot/dts/qcom/msm8996.dtsi
+++ b/arch/arm64/boot/dts/qcom/msm8996.dtsi
@@ -1507,6 +1507,30 @@ cdc_reset_sleep: cdc-reset-sleep {
 				output-low;
 			};
 
+			blsp1_spi5_default: blsp1-spi5-default {
+				spi {
+					pins = "gpio81", "gpio82", "gpio84";
+					function = "blsp_spi5";
+					drive-strength = <12>;
+					bias-disable;
+				};
+
+				cs {
+					pins = "gpio83";
+					function = "gpio";
+					drive-strength = <16>;
+					bias-disable;
+					output-high;
+				};
+			};
+
+			blsp1_spi5_sleep: blsp1-spi5-sleep {
+				pins = "gpio81", "gpio82", "gpio83", "gpio84";
+				function = "gpio";
+				drive-strength = <2>;
+				bias-pull-down;
+			};
+
 			blsp2_spi6_default: blsp2-spi5-default {
 				spi {
 					pins = "gpio85", "gpio86", "gpio88";
@@ -3093,6 +3117,23 @@ blsp1_i2c5: i2c@757a000 {
 			status = "disabled";
 		};
 
+		blsp1_spi5: spi@7579000 {
+			compatible = "qcom,spi-qup-v2.2.1";
+			reg = <0x07579000 0x600>;
+			interrupts = <GIC_SPI 99 IRQ_TYPE_LEVEL_HIGH>;
+			clocks = <&gcc GCC_BLSP1_QUP5_SPI_APPS_CLK>,
+				 <&gcc GCC_BLSP1_AHB_CLK>;
+			clock-names = "core", "iface";
+			pinctrl-names = "default", "sleep";
+			pinctrl-0 = <&blsp1_spi5_default>;
+			pinctrl-1 = <&blsp1_spi5_sleep>;
+			dmas = <&blsp1_dma 20>, <&blsp1_dma 21>;
+			dma-names = "tx", "rx";
+			#address-cells = <1>;
+			#size-cells = <0>;
+			status = "disabled";
+		};
+
 		blsp2_dma: dma@7584000 {
 			compatible = "qcom,bam-v1.7.0";
 			reg = <0x07584000 0x2b000>;
--

Now to enable it in the device DTS:

&blsp1_spi5 {
	status = "okay";
};

Enabling this SPI master ends up freezing the device as soon as the its driver probes and tries to initialize it. But why?

Perhaps taking a closer look at how fingerprint authentication is handled in Android would provide some answers as to why this happened.

Fingerprint sensors in Android

Fingerprint sensors are generally managed in Android’s Hardware Abstraction Layer, or HAL, which mostly consists of proprietary software provided by the vendor. The HAL has some sort of daemon which communicates with the Trusted Execution Environment (TEE for short), where all the real magic happens. The daemon loads a trustlet—a small signed program into the TEE where it is executed. This trustlet includes the actual driver for the fingerprint sensor, and is what scans and matches fingerprints. The daemon would request authentication from the trustlet, which would then scan and match a fingerprint in the secure world, and send a message back to the normal world indicating whether authentication was successful or not. So the real work happens in the secure world, and the normal world only gets to know whether to grant or deny access.

TEE? Secure world? Normal world? What?

To put simply, ARM CPUs have a hardware mechanism known as TrustZone that allows for running two operating systems in parallel, one being the normal OS (Linux in this case) running in the normal world, as well as a trusted OS running in the secure world and providing the system with a trusted execution environment. The normal world’s bus and memory access can be restricted so that select busses and memory ranges become only accessible in the secure world. This is the case with the SPI master that we tried to access from Linux earlier, the access of which was restricted to the secure world. Trying to access it from the normal world caused an exception which halted the execution of Linux. The trusted OS might have printed some error messages over debug UART when that happened, but it is unfortunately inaccessible on this production device, at least without some delicate hardware modifications, so there is no way to confirm that at the moment.

What can be done?

It would seem that the only option to make use of the fingerprint sensor on Linux would be to do it the Android way. Similar to other TEE implementations, the Qualcomm Secure Execution Environment – or QSEE – has a mechanism to handle communications with the normal OS called QSEECOM, short for QSEE COMmunicator, which the mainline Linux kernel is lacking a driver for. Writing a driver in the mainline kernel for QSEECOM should be possible thanks to the availability of downstream driver source code, but another driver, possibly in userspace, will be needed to replace the Android HAL daemon and handle loading and communicating with the proprietary fingerprint sensor trustlet provided by the vendor. Unfortunately, it would be quite difficult to make this second driver due to the lack of open source code or documentation.

The end?

Could this really be a dead end? Or is there still something that can be done? Backtracking a bit might help us find a way.

The SPI master inside the SoC connects to the fingerprint sensor through a set of GPIO pins. These pins are also usually secured on Android phones, making it a necessary step when porting the mainline kernel to a phone with a fingerprint sensor to mark them in the device tree as reserved GPIOs, so that the pin controller driver does not try to initialize them or modify their states and end up causing something similar to what happened after enabling the secured SPI master. Somehow, the Mi Note 2 can run the mainline kernel just fine without reserving any pins. Could this actually mean that the pins the fingerprint sensor is wired to are not secured on this device?

Skipping the hardware SPI master

If the only thing that is secured is the SPI master, then it should be possible to simply skip it all together and use the GPIO pins directly using bit banging to form a software-based SPI master. Software SPI is much slower than its hardware counterpart, but it should still be more than enough for a fingerprint sensor.

Linux has a spi-gpio driver that can do this, and all that is needed to use it is to add a node to the device tree, specifying the GPIO pins to use for SPI. In this case, the four GPIO pins originally used by SPI 5 are used:

	spi {
		compatible = "spi-gpio";

		mosi-gpios = <&tlmm 81 0>;
		miso-gpios = <&tlmm 82 0>;
		cs-gpios = <&tlmm 83 GPIO_ACTIVE_LOW>;
		sck-gpios = <&tlmm 84 0>;
		num-chipselects = <1>;
	};

Booting with this added to the device tree caused no issues. This does not mean it will necessarily work, but is a good sign nevertheless. Now we can proceed to write a driver for the fingerprint sensor.

Writing a driver

$ ls linux/drivers
accessibility  char         dio       greybus     interconnect  memory    nvdimm    pnp         rtc        target       virtio
acpi           clk          dma       hid         iommu         memstick  nvme      power       s390       tc           visorbus
amba           clocksource  dma-buf   hsi         ipack         message   nvmem     powercap    sbus       tee          vlynq
android        comedi       edac      hv          irqchip       mfd       of        pps         scsi       thermal      vme
ata            connector    eisa      hwmon       isdn          misc      opp       ps3         sh         thunderbolt  w1
atm            counter      extcon    hwspinlock  Kconfig       mmc       parisc    ptp         siox       tty          watchdog
auxdisplay     cpufreq      firewire  hwtracing   leds          most      parport   pwm         slimbus    uio          xen
base           cpuidle      firmware  i2c         macintosh     mtd       pci       rapidio     soc        usb          zorro
bcma           crypto       fpga      i3c         mailbox       mux       pcmcia    ras         soundwire  vdpa
block          cxl          fsi       idle        Makefile      net       perf      regulator   spi        vfio
bluetooth      dax          gnss      iio         mcb           nfc       phy       remoteproc  spmi       vhost
bus            dca          gpio      infiniband  md            ntb       pinctrl   reset       ssb        video
cdrom          devfreq      gpu       input       media         nubus     platform  rpmsg       staging    virt

Wait, where do fingerprint sensor drivers go?

It appears that the Linux kernel does not have a fingeprint subsystem, and the only way to use a fingeprint sensor at the moment is through userspace drivers, most of which are part of libfprint. While userspace drivers might be good enough for polled USB sensors, they would not be enough for this device where handling interrupts and regulators is necessary, something that is not easily done in userspace. Libfprint has a few drivers for SPI sensors, but they are all made for ACPI platforms where the driver does not have to worry about regulators and can simply rely on ACPI to do the work. The current aim is to find out if it is possible at all to communicate with the sensor from Linux, so for now, a small SPI class driver placed anywhere in the kernel should be enough.

Collecting information

As always, the main sources of information about the hardware are datasheets in first place, followed by downstream driver source code. A quick search for FPC1020 reveals source code of a downstream driver. Not only that, but this repository happens to also include a datasheet for FPC1140 which is part of the FPC1020 series. Perfect.

The datasheet includes a register map containing information about all the available registers on the sensor. This register looks like a good starting point: If the sensor is working, reading this register would succeed, and the read value should match the format shown in the datasheet.

Writing the initial driver

Being a temporary driver, diving into the details at this point is unnecessary. Initialization steps and other parts such as the fpc1020_access_reg function used to handle SPI transfers needed for accessing registers are not covered here. The final revision of this temporary driver can be found here.

Now, to read the hwID register:

	error = fpc1020_access_reg(chip, FPC102X_REG_HWID, FPC102X_REG_HWID_SIZE,
				&hw_id, false);
	if (error) {
		dev_err(chip->dev, "Failed to read HW ID: %d\n", error);
		return error;
	}
	dev_info(chip->dev, "Chip type: FPC1%x%X\n", hw_id >> 4, hw_id & 0xf);

Here comes the moment of truth:

[   50.284353] fpc1020 spi0.0: Chip type: FPC1140C

No crashes, no freezes and no errors so far. Looks like it worked, and now we know that this phone has a FPC1140 revision C.

In fact, after adding some extra code, a fingerprint can be captured and printed as kernel messages, a tiny section of which can be seen in the cover image of this post.

So these four GPIO pins are definitely not secured. But why? Was it a workaround for a last-minute bug that the engineers at Xiaomi did not have the time to fix properly? Did someone simply forget to secure them? Perhaps we will never know. This also seems to be the case on the Mi 5, another Xiaomi device with the MSM8996 SoC which was able to boot the mainline kernel without reserving any pins. Following this pattern, this might also be the case in the remaining three devices of the Xiaomi MSM8996 platform: Mi 5s, Mi Mix and Mi 5s Plus.

The existence of a downstream driver for this fingeprint sensor is also interesting, as it could mean that there are Android phones where the fingerprint sensor is driven by Linux in the normal world by default.

What comes next

Now that a way to communicate with the sensor is found, it is time to write a proper driver, and eventually be able to authenticate using a fingerprint. As mentioned earlier, Linux has no fingerprint subsystem, and this sensor will require handling regulators and interrupts that must be done in kernel space. This leaves us with two options: either to write a small kernel driver only to handle regulators and somehow also deal with interrupts, then do the rest in userspace, or to make a fingerprint subsystem, which would provide a base to support this and other fingerprint sensors in the kernel. This choice will, however, have to wait for the next part of this series.

Enabling hardware navigation buttons on Linux

Yassine Oudjana — Wed, 27 Oct 2021 00:00:00 +0000

The Xiaomi Mi Note 2 is old enough to (fortunately) have hardware navigation buttons. Let us see what it takes to make them work on the mainline Linux kernel.

This is the first driver I ever wrote. The simplicity of I²C input devices, especially one with just 2 buttons made it a perfect entry point into driver development.

Collecting information

Usually the first step in enabling hardware on the mainline kernel is to take a look at the stock device tree. For the un-initiated, a Device Tree is a method of describing hardware. The OS reads a device tree to find out what hardware is available on the device it is running on and how it is wired up so that it can properly operate it.

The chip that controls hardware navigation buttons is usually called a touchkey controller. Many devices combine that functionality into the touchscreen, placing the navigation buttons in a specific zone, then treating touches in that zone as key presses.

However, that does not seem to be the case in this device.

Looking at the stock device tree, this node is found:

cyttsp_streetfighter_p2@28 {
	compatible = "cypress,sf3155";
	reg = <0x28>;
	interrupt-parent = <&tlmm>;
	interrupts = <77 0x2002>;
	cyttsp,soft-reset;
	cyttsp,irq-gpio = <&tlmm 78 0x2002>;
	cyttsp,irqflags = <0x2002>;
	cyttsp,input-name = "cyttsp_button";
	cyttsp,key-num = <2>;
	cyttsp,key-codes = <139 158>;
	cyttsp,button-status-reg = <0x4A>;
	cyttsp,bootloader-addr = <0x56>;
	cyttsp,config-array-size = <1>;
	cyttsp,standby-reg = <0x01>;
	cyttsp,softreset-reg = <0x05>;
	pinctrl-names = "pmx_btn_active", "pmx_btn_suspend";
	pinctrl-0 = <&btn_active_a4>;
	pinctrl-1 = <&btn_suspend_a4>;
	cyttsp,cfg_1 {
		cyttsp,hw-version = <0x84>;
		cyttsp,fw-name = "cyttsp_button_no_shielding.fw";
	};
};

It is likely to be the touchkey controller; since it is on the same I²C bus as the touchscreen. This line, however, makes it certain:

cyttsp,key-codes = <139 158>;

Looking at linux-event-codes.h, these key codes translate to KEY_MENU and KEY_BACK, which are the functions of the navigation buttons.

Using the compatible string, the driver that matches this node in the downstream kernel can be found:

drivers/input/touchscreen/cyttsp_button.c:1561:

static struct of_device_id cyttsp_match_table[] = {
	{ .compatible = "cypress,sf3155",},
	{ },
};

Cypress Streetfighter

Cool name, right? You don’t see components named like this everyday…

Writing a driver

We start off with the general driver structure, which is similar across all kinds of drivers:

struct device_data {
	...
};

static int device_probe(struct class_device *cdev)
{
	...
}

static int device_remove(struct class_device *cdev)
{
	...
}

static struct of_device_id id_table[] = {
	{ .compatible = "vendor,device" },
	{ /* sentinel */ }
};
MODULE_DEVICE_TABLE(of, id_table);

static struct class_driver device_driver = {
	.driver {
		.name = "driver_name",
		.of_match_table = id_table
	}
	.probe = device_probe,
	.remove = device_remove
};
module_class_driver(device_driver);

Now to put it in more detail:

struct device_data

This is an optional struct. It is conventionally used to hold all information about the device, usually collected during probing, and pass it to functions in the driver.

static int device_probe(struct class_device *cdev)

This is the probe function. It is the first function called in any driver, and is used to allocate resources for the driver, collect information about the hardware and initialize it.

static int device_remove(struct class_device *cdev)

An optional function used to deinitialize the hardware if needed, as well as free any resources that have to be explicitly freed.

static struct of_device_id id_table[] = {
	{ .compatible = "vendor,device" },
	{ /* sentinel */ }
};
MODULE_DEVICE_TABLE(of, id_table);

This table contains compatible strings used to match nodes in the device tree with the driver. When Linux finds a node with a matching compatible string, it loads an instance of the driver and passes the device tree node to it.

static struct class_driver device_driver = {
	.driver {
		.name = "driver_name",
		.of_match_table = id_table
	}
	.probe = device_probe,
	.remove = device_remove
};
module_class_driver(device_driver);

This last struct holds information about the driver, as well as pointers to the probe and remove functions. it is used to register the driver.

There are several driver classes, which are part of the multiple subsystems of the kernel. Each one has a slightly different driver structure. The main differences are marked with the class placeholder above.

The Linux Kernel Documentation is quite useful when it comes to writing a driver. It always helps to read about the subsystems and frameworks needed for a specific device when writing a driver for it. For this device, the Input and I²C subsystems are the ones of interest to us:

This driver will follow the structure shown in the Implementing I²C device drivers page, and use bits from the Creating an input device driver page as needed.

First steps

The first step is to define a data struct to hold information that has to be moved around. Initially, this will hold pointers to the i2c_client and input_dev of this device. It will be later expanded as needed.

struct cypress_sf_data {
	struct i2c_client *client;
	struct input_dev *input_dev;
};

Now, a probe function is added:

static int cypress_sf_probe(struct i2c_client *client,
				const struct i2c_device_id *id)
{
	return 0;
}

This will contain everything required to acquire resources for the driver as well as to initialize the hardware.

First, memory is allocated for a struct cypress_sf_data. This struct will be used throughout the life of the driver.

	struct cypress_sf_data *touchkey;

	touchkey = devm_kzalloc(&client->dev, sizeof(*touchkey), GFP_KERNEL);
	if (!touchkey)
		return -ENOMEM;

devm_kzalloc is more or less the kernel equivalent of calloc. It allocates memory and initializes it to zero. This function can fail and return NULL, in which case the -ENOMEM error code must be returned to signal inability to allocate memory

Now the struct can be filled with data. A pointer to the I²C client is kept for later use:

	touchkey->client = client;
	i2c_set_clientdata(client, touchkey);

i2c_set_clientdata stores a pointer in client to touchkey, which can later be retrieved using i2c_get_clientdata. This is useful in functions where touchkey cannot be passed as an argument, but i2c_client is passed.

With all of this out of the way, we can start to focus on initializing the hardware.

Unfortunately, datasheets are nowhere to be found, so the only sources of information about this touchkey controller are the downstream device tree and driver written by Xiaomi.

Analyzing the downstream driver

The probe function of the downstream cyttsp-button driver should contain everything we need to know about initializing the touchkey controller.

It starts off by initializing a data struct, similar to our driver:

	if (client->dev.of_node) {
		pdata = devm_kzalloc(&client->dev,
				sizeof(struct cyttsp_button_platform_data), GFP_KERNEL);
		if (!pdata) {
			dev_err(&client->dev, "Failed to allocate memroy for pdata\n");
			return -ENOMEM;
		}

Then, it parses information from the device tree:

		error = cyttsp_button_parse_dt(&client->dev, pdata);
		if (error)
			return error;

Parsing the device tree node

Looking at cyttsp_button_parse_dt, the main device tree properties are found. It starts off by reading the Interrupt Request (IRQ) GPIO pin:

	pdata->irq_gpio = of_get_named_gpio_flags(np, "cyttsp,irq-gpio",
				0, &pdata->irq_gpio_flags);

This is the matching property in the device tree:

	cyttsp,irq-gpio = <&tlmm 78 0x2002>;

When a change occurs in the state of the buttons, this pin becomes active and triggers an interrupt to handle the change.

It then goes on to read other properties such as cyttsp,input-name, cyttsp,cut-off-power and cyttsp,soft-reset. These properties are not essential so they can be skipped for now.

After that, it reads addresses of a few registers:

	ret = of_property_read_u32(np, "cyttsp,button-status-reg",
			&temp_val);
	...

	ret = of_property_read_u32(np, "cyttsp,standby-reg",
			&temp_val);
	...
	if (pdata->soft_reset) {
		ret = of_property_read_u32(np, "cyttsp,softreset-reg",
				&temp_val);
		...

Getting button status is done by reading the status register, so this register is important.

Next, it reads a bootloader address:

	ret = of_property_read_u32(np, "cyttsp,bootloader-addr",
			&temp_val);
	if (ret)
		dev_err(dev, "Unable to read bootloader address\n");
	else
		pdata->bootloader_addr = (u8)temp_val;

This touchkey controller has some on-chip flash memory to store firmware, and this address is used in a firmware flashing routine. The chip comes with firmware preinstalled and there is no need to update it, so this is not necessary.

Finally, It reads information about the available keys:

	ret = of_property_read_u32(np, "cyttsp,key-num", &temp_val);
	if (ret) {
		dev_err(dev, "Unable to read key num\n");
		return ret;
	} else
		pdata->nbuttons = temp_val;

	if (pdata->nbuttons != 0) {
		pdata->key_code = devm_kzalloc(dev,
					sizeof(int) * pdata->nbuttons, GFP_KERNEL);
		if (!pdata->key_code)
			return -ENOMEM;
		ret = of_property_read_u32_array(np, "cyttsp,key-codes",
						pdata->key_code, pdata->nbuttons);
		if (ret) {
			dev_err(dev, "Unable to read key codes\n");
			return ret;
		}
	}

Here it gets the number of keys available and their codes. This is important, so it will be implemented in our driver.

It also reads a cyttsp,config-array-size property, which has the number of available configurations. It seems that muliple configurations can be used with a set of firmware files, by loading the corresponding file for the needed configuration. This device only has one configuration, so we do not have to worry about this.

Regulators

Once it is done parsing the device tree node, cyttsp_button_probe calls cyttsp_initialize_regulator. These lines summarize the function:

	data->regulator_vdd = devm_regulator_get(&client->dev, "vdd");

	data->regulator_avdd = devm_regulator_get(&client->dev, "avdd");

	ret = regulator_enable(data->regulator_vdd);

	ret = regulator_enable(data->regulator_avdd);

It gets two regulators: a digital supply vdd and an analog supply avdd. These regulators supply the digital and analog parts of the IC. It then enables them to power on the chip.

After that, it invokes a firmware update routine. As said before, this is not necessary for now.

Interrupts

With regulators initialized, it goes on to request a GPIO pin for an interrupt:

	error = gpio_request(pdata->irq_gpio, "cyttsp_button_irq_gpio");

Then sets its direction:

	error = gpio_direction_input(pdata->irq_gpio);

A while later, it allocates an IRQ using the GPIO pin it got:

	error = request_threaded_irq(client->irq, NULL, cyttsp_button_interrupt,
					pdata->irqflags, client->dev.driver->name, data);

Now, an interrupt will be triggered whenever a signal is received on the GPIO pin. This allows for handling changes in button states as they happen. A more in-depth look at cyttsp_button_interrupt, which is the handler for this interrupt, will come later.

This amount of information should be enough to complete the probe function of our new driver.

Probing the device

Regulators

Before anything else, the regulators are acquired. A new struct is added to cypress_sf_data:

struct cypress_sf_data {
	...
	struct regulator_bulk_data regulators[2];
};

This struct will hold names of the required supplies. Back to cypress_sf_probe:

	touchkey->regulators[0].supply = "vdd";
	touchkey->regulators[1].supply = "avdd";

Now they can be acquired using devm_regulator_bulk_get:

	devm_regulator_bulk_get(&client->dev,
				ARRAY_SIZE(touchkey->regulators),
				touchkey->regulators);

devm_regulator_bulk_get is a device resource managed (devm) wrapper of regulator_bulk_get that automatically frees resources when they are not needed, such as when this driver instance is destroyed, or when it fails to acquire all resources and returns an error code. That makes it unnecessary to manually free them in a remove function. It takes a pointer to the struct device of this device, number of regulators, and a pointer to the struct regulator_bulk_data filled with names earlier. It may fail and return an error code, so that has to be handled correctly:

	int error;
	...

	error = devm_regulator_bulk_get(&client->dev,
					ARRAY_SIZE(touchkey->regulators),
					touchkey->regulators);
	if (error) {
		dev_err(&client->dev, "Failed to get regulators: %d\n", error);
		return error;
	}

If an error occurs, this will print an error message in the kernel log, then return the error code.

Key codes

Next, the key codes are read from the device tree. struct cypress_sf_data is further expanded with two new elements:

struct cypress_sf_data {
	...
	u32 *keycodes;
	int num_keys;
}

Then those elements are set:

	touchkey->num_keys = device_property_read_u32_array(&client->dev,
						"linux,keycodes", NULL, 0);

device_property_read_u32_array returns the number of elements in an array property when NULL is passed A vendor-specific property name like cyttsp,key-codes should not be used here; since a common linux,keycodes property exists. Common property names are preferred over vendor-specific names (see “DOs and DON’Ts for designing and writing Devicetree bindings”).

Also, a cyttsp,key-num property is not necessary; passing NULL instead of an array makes device_property_read_u32_array return the number of elements in the device tree array.

It can also return an error code, such as when it fails to find the property in the device tree node. Instead of just stopping at it though, it is possible to add a default to fall back to:

	if (touchkey->num_keys < 0) {
		/* Default key count */
		touchkey->num_keys = 2;
	}

With the key code count known, memory is allocated for an array to hold the key codes:

	touchkey->keycodes = devm_kzalloc(&client->dev,
				sizeof(u32) * touchkey->num_keys, GFP_KERNEL);
	if (!touchkey->keycodes)
		return -ENOMEM;

The codes are then read from the device tree through a second call to device_property_read_u32_array, passing touchkey->keycodes instead of NULL this time:

	error = device_property_read_u32_array(&client->dev, "linux,keycodes",
						touchkey->keycodes,
						touchkey->num_keys);

Again, this can fail, in which case a warning message is printed in the kernel log and default key codes are used.

	if (error) {
		dev_warn(&client->dev,
			"Failed to read keycodes: %d, using defaults\n", error);

		/* Default keycodes */
		touchkey->keycodes[0] = KEY_BACK;
		touchkey->keycodes[1] = KEY_MENU;
	}

Powering up the IC

Now would be a good time to power up the chip. That is done by calling regulator_bulk_enable which enables all regulators in the struct regulator_bulk_data passed to it (vdd and avdd in this case).

	error = regulator_bulk_enable(ARRAY_SIZE(touchkey->regulators),
					touchkey->regulators);
	if (error) {
		dev_err(&client->dev, "Failed to enable regulators: %d\n", error);
		return error;
	}

The input subsystem

With most of the initializion done, it is time to register an input device. The Creating an input device driver describes the process in detail, as well as other input-related functions which will be used later. For now, the process will be as follows:

Allocate an input device structure (struct input_dev);
Add some general information about the device to the structure;
Set device capabilites;
Register the input device.

A struct *input_dev was added to struct cypress_sf_data at the beginning. A call to devm_input_allocate_device would allocate an input device and return a pointer to it, which will be stored in the struct *input_dev added to struct cypress_sf_data earlier:

	touchkey->input_dev = devm_input_allocate_device(&client->dev);
	if (!touchkey->input_dev) {
		dev_err(&client->dev, "Failed to allocate input device\n");
		return -ENOMEM;
	}

Then, the name and bus type of the device are set. The device name will be used in other places, so it is better to define it:

#define CYPRESS_SF_DEV_NAME "cypress-sf"

	touchkey->input_dev->name = CYPRESS_SF_DEV_NAME;
	touchkey->input_dev->id.bustype = BUS_I2C;

The third step is to set the capabilities of the input device. This is a touchkey controller, so it has a set of keys, each with its own key code:

	for (key = 0; key < touchkey->num_keys; ++key)
		input_set_capability(touchkey->input_dev, EV_KEY,
					touchkey->keycodes[key]);

This iterates through the key codes acquired from the device tree earlier, and adds a key capability with a key code to the input device.

Finally, the input device is registered:

	error = input_register_device(touchkey->input_dev);
	if (error) {
		dev_err(&client->dev,
			"Failed to register input device: %d\n", error);
		return error;
	}

At this point, the driver is ready to signal input events to the input subsystem, but it still has no way of detecting button state changes. This is where the status interrupt comes into play.

An interrupt line is now allocated for the status interrupt:

	error = devm_request_threaded_irq(&client->dev, client->irq,
					NULL, cypress_sf_irq_handler,
					IRQF_ONESHOT,
					CYPRESS_SF_DEV_NAME, touchkey);
	if (error) {
		dev_err(&client->dev,
			"Failed to register threaded irq: %d", error);
		return error;
	}

devm_request_threaded_irq summarizes what the downstream driver does to get the interrupt line in one function. It reads the interrupt line from the device tree interrupts property, and does everything needed to allocate an IRQ. cypress_sf_irq_handler is the handler function for this interrupt, which will be called every time an IRQ is received.

This sums up the probe function. Now the driver is ready to receive button state changes then signal input events correspondingly, the thing it does in the status interrupt handler.

Handling the status interrupt

The touchkey controller sends an IRQ when any of the keys is pressed or released. This allows the driver to check the button states only when they change. The downstream driver has this interrupt handler:

static irqreturn_t cyttsp_button_interrupt(int irq, void *dev_id)
{
	struct cyttsp_button_data *data = dev_id;
	struct cyttsp_button_platform_data *pdata = data->pdata;
	bool curr_state, new_state;
	bool sync = false;
	u8 val;
	u8 key;
	unsigned long keystates;

	if (data->enable) {
		val = cyttsp_read_reg(data, CYTTSP_REG_TOUCHMODE);
		if (val < 0) {
			dev_err(&data->client->dev, "Failed to read touch mode reg\n");
			return IRQ_NONE;
		} else {
			mutex_lock(&data->input_dev->mutex);
			data->glove_mode = !!(val & (1 << CYTTSP_GLOVE_MODE_SHIFT));
			mutex_unlock(&data->input_dev->mutex);
		}

		val = cyttsp_read_reg(data, pdata->button_status_reg);
		if (val < 0) {
			dev_err(&data->client->dev, "Failed to read status!\n");
			return IRQ_NONE;
		}

		keystates = (unsigned long)val;

		for (key = 0; key < pdata->nbuttons; key++) {
			curr_state = test_bit(key, &data->keystatus);
			new_state = test_bit(key, &keystates);

			if (curr_state ^ new_state) {
				cyttsp_report_key(data, pdata->key_code[key],
						!!(keystates & (1 << key)));
				sync = true;
			}
		}

		data->keystatus = keystates;

		if (sync)
			input_sync(data->input_dev);
	}

	return IRQ_HANDLED;
}

Before doing anything, it checks an enable element in its data struct. Our driver does not have one, so that can be skipped.

Then it checks if the touch mode changed:

		val = cyttsp_read_reg(data, CYTTSP_REG_TOUCHMODE);
		if (val < 0) {
			dev_err(&data->client->dev, "Failed to read touch mode reg\n");
			return IRQ_NONE;
		} else {
			mutex_lock(&data->input_dev->mutex);
			data->glove_mode = !!(val & (1 << CYTTSP_GLOVE_MODE_SHIFT));
			mutex_unlock(&data->input_dev->mutex);
		}

This touchkey controller has a normal mode and a glove mode, possibly for better touch detection with gloved fingertips. Modes are not implemented in our driver as they are not essential, so this part can be skipped too.

Now the important part begins. It first reads the status register, which is 0x4a according to the downstream device tree:

		val = cyttsp_read_reg(data, pdata->button_status_reg);

Then it casts the value it reads to an unsigned long variable named keystates.

The status register stores button states in a bit field, where each bit corresponds to a button. 1 is a pressed button, and 0 is the opposite.

Now it iterates through registered keys (read earlier from the device tree), and gets the old and new states of each. After that, it is just a simple XOR operation to determine which ones changed.

		for (key = 0; key < pdata->nbuttons; key++) {
			curr_state = test_bit(key, &data->keystatus);
			new_state = test_bit(key, &keystates);

			if (curr_state ^ new_state) {
				cyttsp_report_key(data, pdata->key_code[key],
						!!(keystates & (1 << key)));
				sync = true;
			}
		}

Then for each key that had a state change, it calls cyttsp_report_key, and sets sync = true for a reason seen later.

static void cyttsp_report_key(struct cyttsp_button_data *data,
		int key, int status)
{
	if (key == KEY_MENU)
		input_report_key(data->input_dev,
				data->enable_reversed_keys ? KEY_MENU : KEY_BACK, status);
	else if (key == KEY_BACK)
		input_report_key(data->input_dev,
				data->enable_reversed_keys ? KEY_BACK : KEY_MENU, status);
	else
		input_report_key(data->input_dev, key, status);
}

This function is just a wrapper for input_report_key, with the sole purpose of allowing the back and menu buttons to be swapped. There should be better ways to implement this – possibly in userspace – so it will not be implemented it in our driver.

Finally, it stores the new states to use as old states in the next status interrupt,

		data->keystatus = keystates;

then calls input_sync, which makes the input subsystem send input events according to the previous calls to input_report_key. It uses the sync variable from earlier to check if the status had changed:

		if (sync)
			input_sync(data->input_dev);

Writing an interrupt handler

Using the previous findings, a new interrupt handler is written.

static irqreturn_t cypress_sf_irq_handler(int irq, void *devid)
{
	...

	return IRQ_HANDLED;
}

Interrupt handlers have an irqreturn_t return type, which is an enum with a few entries:

/**
 * enum irqreturn
 * @IRQ_NONE		interrupt was not from this device or was not handled
 * @IRQ_HANDLED		interrupt was handled by this device
 * @IRQ_WAKE_THREAD	handler requests to wake the handler thread
 */
enum irqreturn {
	IRQ_NONE		= (0 << 0),
	IRQ_HANDLED		= (1 << 0),
	IRQ_WAKE_THREAD		= (1 << 1),
};

typedef enum irqreturn irqreturn_t;

and as arguments, they take int irq, the interrupt line that signaled the IRQ, and a void *dev_id, which can be used to pass data to the interrupt handler. These arguments correspond to what was passed when allocating the interrupt, which were client->irq and the data struct touchkey.

As such, void *devid can be casted to a struct cypress_sf_data *:

	struct cypress_sf_data *touchkey = devid;

Reading the status register

The I²C subsystem provides a function to read a byte from a register: i2c_smbus_read_byte_data. It takes the I²C client and the register address as arguments, and returns the read value, or a negative error code when it fails.

The register address will be defined in the driver instead of the device tree this time:

#define CYPRESS_SF_REG_BUTTON_STATUS	0x4a

Then i2c_smbus_read_byte_data can be used to read the status register, and store the value in an int val variable:

	int val;

	val = i2c_smbus_read_byte_data(touchkey->client,
					CYPRESS_SF_REG_BUTTON_STATUS);

As usual, errors must be handled properly. Since this is an interrupt handler, IRQ_NONE must be returned after an error to signal the inability to handle the interrupt:

	if (val < 0) {
		dev_err(&touchkey->client->dev, "Failed to read button status: %d",
			val);
		return IRQ_NONE;
	}

val can then be casted to an unsigned long keystates for later operations:

	unsigned long keystates;
	...

	keystates = val;

(I am not quite sure if this is necessary, but it is what I did in the original driver, so I will proceed with it)

Finding status changes

Similar to the downstream driver, an XOR operation is used to find which button states changed. For this, keeping track of the status history is needed, so a keystates element is added to struct cypress_sf_data to hold the current key states (which become the old key states when the status interrupt handler is called):

struct cypress_sf_data {
	...
	unsigned long keystates;
}

Then, an unsigned long changed is defined to hold the XOR result:

unsigned long keystates, changed;

Instead of using the XOR operator (^), bitmap_xor can be used:

	bitmap_xor(&changed, &keystates, &touchkey->keystates,
		   touchkey->num_keys);

This will perform an XOR operation between keystates and touchkey->keystates, or in other words, the new and old states. The result will then be stored in changed.

Next, using for_each_set_bit to iterate through each bit (or key state), the new state is reported using input_report_key:

	for_each_set_bit(key, &changed, touchkey->num_keys) {
		new_state = keystates & BIT(key);
		dev_dbg(&touchkey->client->dev,
				"Key %d changed to %d", key, new_state);
		input_report_key(touchkey->input_dev,
				touchkey->keycodes[key],
				new_state);
	}

For debugging purposes, a debug message is added as well.

Finally, input_sync is called to synchronize the states with the input subsystem and let it send input events:

	input_sync(touchkey->input_dev);

Then the new key states are stored in the keystates element of the data struct to be used in the next status interrupt:

	touchkey->keystates = keystates;

With that, the driver becomes mostly complete.

Final touches

Adding a kconfig entry and a line to the Makefile is needed to build it. That and other additions such as power managment callbacks and a device tree schema are included in the final patchset.

Enabling the touchkey controller

With a driver written, all it would take to make the navigation buttons work is to add a device tree node.

The touchkey controller is connected to the 6^th I²C bus of BLSP 2 on the MSM8996 SoC, so a node is added there:

&blsp2_i2c6 {
	touchkeys: cypress-sf@28 {
		compatible = "cypress,sf3155";
		reg = <0x28>;
	};
};

compatible is the same string added to the match table in the driver, while reg, which specifies the I²C address, is taken from the downstream device tree.

Next, the status interrupt is added:

		interrupt-parent = <&tlmm>;
		interrupts = <77 IRQ_TYPE_EDGE_FALLING>;

interrupt-parent specifies the interrupt controller that provides this interrupt, which in this case is TLMM, the GPIO controller. interrupts specify the interrupt lines used, and their types. Here, 77 resembles GPIO pin 77, while IRQ_TYPE_EDGE_FALLING makes it so that the interrupt is triggered in the falling edge of the signal, or in other words, during the transition from high to low.

Next, the digital and analog power supplies are added:

		avdd-supply = <&vreg_l6a_1p8>;
		vdd-supply = <&vdd_3v2_tp>;

avdd is powered by l6, which is a Low-dropout regulator, or LDO for short, built into the Power Managment IC (PMIC). Meanwhile, vdd is powered by a separate LDO.

Finally, the key codes are specified using the linux,keycodes property:

		linux,keycodes = <KEY_BACK KEY_MENU>;

This commit adds the device tree node, in addition to pin states used for suspend/resume

And just like that, we get working navigation buttons on Linux.

Unfortunately those buttons are not utilized much in userspace yet. The only things I found that react to pressing them are Firefox, which uses the back key to – you guessed it – go back, and GNOME Control Center, where pressing the back key causes some strange behavior, making it randomly jump to the power or network section possibly due to a bug. This, however, is a topic for another post.

Unlocking the Qualcomm Snapdragon Sensor Core (Part 1)

Yassine Oudjana — Wed, 06 Oct 2021 00:00:00 +0000

Sensors have generally been some of the easier components to support on the mainline Linux kernel; considering that on most devices, all it takes to enable a sensor is to enable the I²C or SPI interface it is connected to, describe it in the device tree, and in the worst case scenario, write a driver for it with the help of detailed, publicly available datasheets.

This, however, became longer the case in devices using Qualcomm SoCs. With the release of the Snapdragon 820, the Snapdragon Sensor Core (SSC) – sometimes also referred to as the Sensor Low Power Island (SLPI) – was introduced. It is a sensor hub consisting of GPIOs that provide I²C, SPI and serial interfaces to receive data from sensors, and a Hexagon core to process the data. It was introduced as a power-saving measure which reduces power consumption by offloading the processing of sensor data to a low-power DSP, allowing the more power-hungry CPUs to work less and even be put to deeper sleep states when idle.

Unfortunately, this means that handling sensors is entirely done in proprietary firmware which communicates with a proprietary HAL driver on Android. With no source code nor documentation, we are left with nothing to work with on Linux. What makes matters worse is that there is no way, at least on consumer devices, to bypass SSC entirely and directly access the GPIOs to which the sensors are connected. If this was possible, it would have at least allowed for driving them similar to older devices, albeit with a degradation in idle battery life.

This series will go through the process of reverse engineering the proprietary driver for SSC on Snapdragon 820, and in the end, hopefully writing an open source driver to make use of SSC on Linux.

I will be testing on the Xiaomi Mi Note 2 in this series.

The model name of Snapdragon 820 is MSM8996, and as so it will be referred to as that throughout this series.

First steps

Before doing any actual reversing, it must be made sure that SLPI can be booted on Linux. The qcom_q6v5_pas kernel driver handles loading firmware into and booting QDSP6 – or Hexagon – remote processors such as SLPI and ADSP. The firmware is first loaded into a predefined region of memory and authenticated in the secure world, then the remote processor gets reset and starts executing it. These memory regions should be reserved so that Linux does not map them, and leaves them to be exclusively used by the remote processor and the driver that loads its firmware. These are the reserved memory regions originally defined in the MSM8996 device tree:

reserved-memory {
	#address-cells = <2>;
	#size-cells = <2>;
	ranges;

	mba_region: mba@91500000 {
		reg = <0x0 0x91500000 0x0 0x200000>;
		no-map;
	};

	slpi_region: slpi@90b00000 {
		reg = <0x0 0x90b00000 0x0 0xa00000>;
		no-map;
	};

	venus_region: venus@90400000 {
		reg = <0x0 0x90400000 0x0 0x700000>;
		no-map;
	};

	adsp_region: adsp@8ea00000 {
		reg = <0x0 0x8ea00000 0x0 0x1a00000>;
		no-map;
	};

	mpss_region: mpss@88800000 {
		reg = <0x0 0x88800000 0x0 0x6200000>;
		no-map;
	};

	smem_mem: smem-mem@86000000 {
		reg = <0x0 0x86000000 0x0 0x200000>;
		no-map;
	};

	memory@85800000 {
		reg = <0x0 0x85800000 0x0 0x800000>;
		no-map;
	};

	memory@86200000 {
		reg = <0x0 0x86200000 0x0 0x2600000>;
		no-map;
	};

	rmtfs@86700000 {
		compatible = "qcom,rmtfs-mem";

		size = <0x0 0x200000>;
		alloc-ranges = <0x0 0xa0000000 0x0 0x2000000>;
		no-map;

		qcom,client-id = <1>;
		qcom,vmid = <15>;
	};

	zap_shader_region: gpu@8f200000 {
		compatible = "shared-dma-pool";
		reg = <0x0 0x90b00000 0x0 0xa00000>;
		no-map;
	};
};

Taking a closer look at slpi_region and zap_shader_region, it seems that somehow they ended up having the same address:

slpi_region: slpi@90b00000 {
	reg = <0x0 0x90b00000 0x0 0xa00000>;
	no-map;
};

zap_shader_region: gpu@8f200000 {
	compatible = "shared-dma-pool";
	reg = <0x0 0x90b00000 0x0 0xa00000>;
	no-map;
};

This will cause problems, as both the GPU and SLPI drivers will try to load firmware into the same region. Fixing this is required before doing anything about SLPI.

Looking at the downstream kernel, there are no specific memory regions for each remote processor, but rather one big shared region:

peripheral_mem: peripheral_region@8ea00000 {
	compatible = "removed-dma-pool";
	no-map;
	reg = <0 0x8ea00000 0 0x2d00000>;
};

Fortunately though, the downstream drivers report the memory ranges they load firmware into in the kernel log:

# dmesg | grep "loading from"
[   10.272396] subsys-pil-tz 1c00000.qcom,ssc: slpi: loading from 0x000000008fe00000 to 0x0000000090800000
[   10.273749] subsys-pil-tz 9300000.qcom,lpass: adsp: loading from 0x0000000090800000 to 0x0000000092300000
[   11.229705] pil-q6v5-mss 2080000.qcom,mss: modem: loading from 0x0000000089c00000 to 0x000000008fe00000
[   11.751973] subsys-pil-tz soc:qcom,kgsl-hyp: a530_zap: loading from 0x0000000092300000 to 0x0000000092302000
[   26.233731] subsys-pil-tz ce0000.qcom,venus: venus: loading from 0x0000000092400000 to 0x0000000092900000

Comparing those ranges to peripheral_mem above, it is found that they are actually shifted forward by 0x1400000 on this device, and this is further confirmed by taking a quick look at the device DTS, where the original region defined in the SoC DTS is overwritten:

/delete-node/ peripheral_region@8ea00000;
peripheral_mem: peripheral_region@8fe00000 {
	compatible = "removed-dma-pool";
	no-map;
	reg = <0x0 0x8fe00000 0x0 0x2b00000>;
};

Putting all of this information together gives the final regions:

Region	Address	Size
MPSS	`0x88800000`	`0x6200000`
ADSP	`0x8ea00000`	`0x1b00000`
SLPI	`0x90500000`	`0xa00000`
GPU (Zap Shader)	`0x90f00000`	`0x100000`
Venus	`0x91000000`	`0x500000`
MBA	`0x91500000`	`0x200000`

This patch takes those regions and defines them in the SoC DTS, and updates the shifted ranges in the device DTS to match.

With reserved memory taken care of, now we can start working on booting SLPI.

Adding SLPI to the device tree

Following the device tree schema for qcom_q6v5_pas, we can start writing a node for SLPI, with reg taken from the downstream device tree:

slpi_pil: remoteproc@1c00000 {
	compatible = "qcom,msm8996-slpi-pil";
	reg = <0x01c00000 0x4000>;
};

Interrupts

There are five interrupts in total: One real interrupt belonging to the global interrupt controller used for a watchdog signal, and four virtual SMP2P (Shared Memory Point to Point) interrupts used to signal various events. Details about shared memory will come later.

slpi_pil: remoteproc@1c00000 {
...
	interrupts-extended = <&intc 0 390 IRQ_TYPE_EDGE_RISING>,
			<&slpi_smp2p_in 0 IRQ_TYPE_EDGE_RISING>,
			<&slpi_smp2p_in 1 IRQ_TYPE_EDGE_RISING>,
			<&slpi_smp2p_in 2 IRQ_TYPE_EDGE_RISING>,
			<&slpi_smp2p_in 3 IRQ_TYPE_EDGE_RISING>;
	interrupt-names = "wdog",
			"fatal",
			"ready",
			"handover",
			"stop-ack";
};

Clocks

There are two clocks driving SLPI: One is the on-board oscillator (xo_board), and the other is the bus clock of the NoC (Network-on-Chip) SLPI is connected to (aggre2):

slpi_pil: remoteproc@1c00000 {
...
	clocks = <&xo_board>,
		<&rpmcc RPM_SMD_AGGR2_NOC_CLK>;
	clock-names = "xo", "aggre2";
};

Memory region

A reference to the reserved memory region defined earlier is needed to load firmware into:

slpi_pil: remoteproc@1c00000 {
	...
	memory-region = <&slpi_mem>;
};

Shared memory

Qualcomm SoCs make use of a region in memory that is shared between multiple processors on the SoC, through which the CPU – usually referred to as the AP, or Application Processor – can communicate with remote processors on the chip. Each processor, or shared memory device (SMD), is called an “edge”, and has a unique identifier smd-edge, as well as a remote identifier remote-pid used by other processors to refer to it. They also use interrupts and mailbox doorbells to signal events such as opening a SMD channel or receiving a message.

remote-pid has alredy been added to the mainline DTS in the SLPI SMP2P node, so it can be simply copied over:

slpi_pil: remoteproc@1c00000 {
	...
	smd-edge {
		qcom,remote-pid = <3>;
	};
};

Searching for dsps which is the SMD label for SLPI, the remaining properties are found:

qcom,smd-dsps {
	compatible = "qcom,smd";
	qcom,smd-edge = <3>;
	qcom,smd-irq-offset = <0x0>;
	qcom,smd-irq-bitmask = <0x2000000>;
	interrupts = <0 176 1>;
	label = "dsps";
};

smd-edge, label, as well as a single interrupt are all clear, so they can be directly added to our SLPI node:

slpi_pil: remoteproc@1c00000 {
	...
	smd-edge {
		interrupts = <GIC_SPI 176 IRQ_TYPE_EDGE_RISING>;

		label = "dsps";
		qcom,smd-edge = <3>;
		qcom,remote-pid = <3>;
	};
};

The mailbox doorbell was not obvious though; it took me a while to figure out that it was actually the position of the enabled bit in smd-irq-bitmask, which is 25 here. Adding that completes the smd-edge subnode:

slpi_pil: remoteproc@1c00000 {
	...
	smd-edge {
		interrupts = <GIC_SPI 176 IRQ_TYPE_EDGE_RISING>;

		label = "dsps";
		mboxes = <&apcs_glb 25>;
		qcom,smd-edge = <3>;
		qcom,remote-pid = <3>;
	};
};

The last thing to add is qcom,smem-states, which are according to the DT schema, “states used by the AP to signal the Hexagon core”. In this case, there is only one state:

slpi_pil: remoteproc@1c00000 {
	...
	qcom,smem-states = <&slpi_smp2p_out 0>;
	qcom,smem-state-names = "stop";

	smd-edge {
		...
	};
};

Final Touches

To finish it off, the node is disabled by default to make SLPI optional since not all devices use it:

slpi_pil: remoteproc@1c00000 {
	...
	status = "disabled";
	...
};

And with that done, the node is complete:

slpi_pil: remoteproc@1c00000 {
	compatible = "qcom,msm8996-slpi-pil";
	reg = <0x01c00000 0x4000>;

	interrupts-extended = <&intc 0 390 IRQ_TYPE_EDGE_RISING>,
			      <&slpi_smp2p_in 0 IRQ_TYPE_EDGE_RISING>,
			      <&slpi_smp2p_in 1 IRQ_TYPE_EDGE_RISING>,
			      <&slpi_smp2p_in 2 IRQ_TYPE_EDGE_RISING>,
			      <&slpi_smp2p_in 3 IRQ_TYPE_EDGE_RISING>;
	interrupt-names = "wdog",
			  "fatal",
			  "ready",
			  "handover",
			  "stop-ack";

	clocks = <&xo_board>,
		 <&rpmcc RPM_SMD_AGGR2_NOC_CLK>;
	clock-names = "xo", "aggre2";

	memory-region = <&slpi_mem>;

	qcom,smem-states = <&slpi_smp2p_out 0>;
	qcom,smem-state-names = "stop";

	status = "disabled";

	smd-edge {
		interrupts = <GIC_SPI 176 IRQ_TYPE_EDGE_RISING>;

		label = "dsps";
		mboxes = <&apcs_glb 25>;
		qcom,smd-edge = <3>;
		qcom,remote-pid = <3>;
	};
};

Enabling SLPI

With the SLPI node added to the SoC DTS, what remains is to enable it in the device DTS and add a firmware path to load device-specific firmware, as well as the PX supply, which can be found in the downstream DTS:

&slpi_pil {
	status = "okay";

	px-supply = <&vreg_lvs2a_1p8>;
	firmware-name = "qcom/msm8996/scorpio/slpi.mbn";
};

Getting firmware

SLPI firmware can be found in /vendor/firmware_mnt/image on this device:

# ls /vendor/firmware_mnt/image | grep slpi
slpi_a1.b00
slpi_a1.b01
slpi_a1.b02
slpi_a1.b03
slpi_a1.b04
slpi_a1.b05
slpi_a1.b06
slpi_a1.b07
slpi_a1.b08
slpi_a1.b09
slpi_a1.b10
slpi_a1.b11
slpi_a1.b12
slpi_a1.b13
slpi_a1.b14
slpi_a1.mdt
slpi_a4.b00
slpi_a4.b01
slpi_a4.b02
slpi_a4.b03
slpi_a4.b04
slpi_a4.b05
slpi_a4.b06
slpi_a4.b07
slpi_a4.b08
slpi_a4.b09
slpi_a4.b10
slpi_a4.b11
slpi_a4.b12
slpi_a4.b13
slpi_a4.b14
slpi_a4.mdt
slpi_a7.b00
slpi_a7.b01
slpi_a7.b02
slpi_a7.b03
slpi_a7.b04
slpi_a7.b05
slpi_a7.b06
slpi_a7.b07
slpi_a7.b08
slpi_a7.b09
slpi_a7.b10
slpi_a7.b11
slpi_a7.b12
slpi_a7.b13
slpi_a7.b14
slpi_a7.mdt

For some reason, Xiaomi decided to ship SLPI firmware for all of their MSM8996 devices on each one of them. The Mi Note 2 is A4 according to the downstream kernel, so the slpi_a4.* files are what we are looking for.

While it is possible to use the firmware in its current segmented form, combining the segments into one file is preferred. This can be done using pil-squasher. The result is a single .mbn file.

Testing

With all of that done, it is time to test everything added so far. If all goes well, a message confirming the successful boot of SLPI should be found in the kernel log.

…But of course, things do not always go as planned.

# dmesg | grep 1c00000
[    6.952588] qcom_q6v5_pas 1c00000.remoteproc: supply cx not found, using dummy regulator
[    7.065288] remoteproc remoteproc0: 1c00000.remoteproc is available
[    7.107279] remoteproc remoteproc0: powering up 1c00000.remoteproc
[    7.189399] qcom_q6v5_pas 1c00000.remoteproc: failed to authenticate image and release reset
[    7.192502] remoteproc remoteproc0: can't start rproc 1c00000.remoteproc: -22

Troubleshooting

Upon further investigation, I found that there is a missing power domain that has to be voted on. Power domains are basically regulators or other resources used by multiple parts of the system usually having different power requirements. Consumers attach to the power domain, then request it to be turned on, and in some cases, set the performance level to fulfill their needs. The power domain then is set to the highest performance level requested in order to statisfy all consumers.

A patch to qcom_q6v5_pas was needed to make it attach to the power domain that powers SLPI. Once that was done, the power domain had to be added to the DTS node:

slpi_pil: remoteproc@1c00000 {
	...
	power-domains = <&rpmpd MSM8996_VDDSSCX>;
	power-domain-names = "ssc_cx";
	...
};

Now SLPI boots successfully:

# dmesg | grep 1c00000
[    6.952588] qcom_q6v5_pas 1c00000.remoteproc: supply cx not found, using dummy regulator
[    7.065288] remoteproc remoteproc0: 1c00000.remoteproc is available
[    7.107279] remoteproc remoteproc0: powering up 1c00000.remoteproc
[   15.117415] remoteproc remoteproc1: remote processor 1c00000.remoteproc is now up

QMI services provided by SLPI can be found using qrtr-lookup:

$ qrtr-lookup
  Service Version Instance Node  Port
  ...
       15       1        8    9     1 Test service
       66       1       20    9     4 Service registry notification service
       43       2       22    9     6 Subsystem control service
       15       1        9    9     7 Test service
      771       1        1    9     9 Peripheral Access Control Manager service

The final patchset also includes bringing up the modem, which is done in a similar way.

With SLPI booting on the mainline kernel, it is time to start communicating with it. Part 2 will cover decoding messages dumped from the proprietary HAL driver in order to understand the way it talks to the remote processor, then attempt to do the same in the open source driver.