Is there any way to directly stream real-time video to remote controller on PSDK?

Completed
Follow
megumu.tsukamoto
  • Edited

How can I use the PSDK to get real-time video stream from a third party camera on DJI Pilot? I would like to know which API or tutorial I should refer to.

#Situation
We are developing gas inspections, and for some reason we need to get real-time video stream from a third-party camera attached to the X-Port and control X-Port.
I can control the X-Port using the PSDK and DJI Pilot, but cannot get live stream from a third party camera on a ground while controlling X-Port.

#What I have tried
I have tested live-view function on PSDK3.0 demo app(dji_sdk_demo_linux).
It seems that the "DjiLiveview_StartH264Stream" function just record the video on aircraft and I cannot see the live stream video by remote controller on the ground.

 

https://sdk-forum.dji.net/hc/zh-cn/articles/900007012123
This article says PSDK can directly transmit the video stream to the remote controller (MSDK) through the DJI wireless link, so I want to see the real-time streaming video of PSDK camera on a ground without any additional telecommunication equipment.

#Device
M300RTK
USB Cam(logicool)
OSDK expansion interface

Third-party development board(raspberry pi4)
PSDK software Development Kit 2.0

#Additional
I want to preprocess the video (overlay the graph of the amount of gas), so I don't mind whether it is OSDK or PSDK as long as I can get real-time video stream of 3rd party cam on the ground and also edit the real-time stream.

Thank you for your daily support.

1

Comments

21 comments

Date Votes
  • megumu.tsukamoto

    #Additional
    I want to preprocess the video (overlay the graph of the amount of gas), so I don't mind whether it is OSDK or PSDK as long as I can get real-time video stream of 3rd party cam on the ground and also edit the real-time stream.

    0
    Comment actions Permalink
  • DJI Developer Support
    Your requirements seems to send the video stream to RC from third party camera, it is PSDK camera emu function, you shuld connects the third party device to the X-PORT or Skyport and mounted on M300. There is a sample about how to send stream(simulator of h264 file) to RC APP。 https://github.com/dji-sdk/Payload-SDK/tree/master/samples/sample_c/module_sample/camera_emu
    1
    Comment actions Permalink
  • megumu.tsukamoto

    Thank you so much for your advice.

    I have confirmed by debugging that I can activate "DjiTest_CameraEmuBaseStartService()" function in sample_c/platform/linux/manifold2/application/main.c, However Pilot App just display front cam video(window name is FPV) and black screen(window name is PSDK_APPALIAS).

    I cannot see the live streamed video on the Pilot app.(I guess PSDK_0005.h264 should have streamed in the demo app)
    How can I see the emulated camera's stream?

    0
    Comment actions Permalink
  • DJI Developer Support
    For this error, it is recommended to ping the drone IP to see if the IP communication works.
    0
    Comment actions Permalink
  • megumu.tsukamoto

    Thank you.
    I did the ping communication with X-port and ip communication seems to works well.

    And I have found another failure which is similar to this article (https://sdk-forum.dji.net/hc/zh-cn/community/posts/4411736973465), that when I use X-port as a PSDK adapter, PSDK menu on Pilot is not fully displayed. (same as 显示实时数据 error)

    PSDK menu on Pilot is fully displayed when I use OSDK expansion board as a PSDK adapter.

    Then what is the problem?

    0
    Comment actions Permalink
  • DJI Developer Support
    Please show the ping screenshot and "ifconfig"
    0
    Comment actions Permalink
  • DJI Developer Support
    Sorry, because the forum will forward the email system to the DJI sdk team, there may be some missing pictures. The picture you provided shows that the network port used is lo (localhost) You can set it as eth0 in the code, and then connect eth0 port to the RJ45 interface of the adapter board through a network cable ![](https://djisdksupport.zendesk.com/attachments/token/cXXzDGkyqTU3Y6vWSrbIenvXn/?name=image.png)
    1
    Comment actions Permalink
  • megumu.tsukamoto

    Thanks to your advice, I can successfully send video file("media_file/PSDK_0005.h264") to DJI pilot by camera_emu function.

    As a next step, I want to send real-time USB Cam stream(like "/dev/video0") to my DJI Pilot, but I can see only black screen in the PSDK menu in Pilot.

    Is this expected behavior due to the fact that the code only supports video files? Or is this the error due to the camera not meeting the payload criterion?

    #How to reproduce

    1. Change "tempPath" name from "%smedia_file/PSDK_0005.h264" to "%smedia_file/video0" in test_payload_cam_emu_base.c.

    2. Change the pixel format of camera by below code because my web cam is not compatible with h264 streaming.

    3. sudo ./dji_sdk_demo_linux

    ffmpeg     -y -f v4l2 -thread_queue_size 8192 -input_format yuyv422     -video_size 640x480 -framerate 30     -i /dev/video0     -c:v h264_v4l2m2m -b:v 768k     -bufsize 768k -vsync 1 -g 10     -f flv /home/ubuntu/PSDK/samples/sample_c/module_sample/camera_emu/media_file/video0

    0
    Comment actions Permalink
  • megumu.tsukamoto

    ##supplement

    Once I save the video stream of USB cam as a mp4 or h264 file about 30 seconds long , then I can send that file to DJI Pilot properly.

    0
    Comment actions Permalink
  • DJI Developer Support
    It is impossible for the sample to simulate real-time capture of video streams, so H264 file transmission is used. Developers need to combine camera sensors to encode H264 video streams, and then send them through the video transmission API of PSDK.
    1
    Comment actions Permalink
  • nmai

    If you look at the test_payload_cam_emu_media.c file in the PSDK sample, the SDK provides a function that accepts video "frames" and transmits it to the remote control.  If using the ffmpeg library, the corresponding class for these "frames" is AVPacket. 
    In other words, to use the PSDK+ffmpeg you will need to go from:
    Video stream => H264 packets => PSDK DjiPayloadCamera_SendVideoStream(h264 packet data + 6-byte marker) => remote control. 
    We have not figured out how to capture packets from a live video stream yet, but it looks like it can be done with ffmpeg library.  The remaining steps are more or less trivial.  Hope this help and if you figure out the capturing packets part, please give us some pointers.

    1
    Comment actions Permalink
  • nmai

    We ended up taking a different approach, but I believe you can use a parser (AVCodecParserContext) to extract packets from a live stream raw data.  See the details in the file decode_video.c from the ffmpeg examples.  A file is used in the sample, but I think the idea should be the same for raw data stream.  

    1
    Comment actions Permalink
  • megumu.tsukamoto

    I've developed the pipeline using FFmpeg+test_payload_cam_emu_media.c file in the PSDK like this ↓.

    Video stream => H264 packets => PSDK DjiPayloadCamera_SendVideoStream(h264 packet data + 6-byte marker) => Remote Control. 

    However, it send black screen to the PSDK menu in the remote controller when I send it in real time. (picture below)

    If once I save the encoded H264 packets to mp4 file and then read it from test_payload_cam_emu_media.c, it successfully send the video to RC.

    Is there anything you know about this phenomenon that would cause a black screen when trying to send a packet directly in real time?

    ## What I did:

    get frame from webcam & encode it → H264 AVPacket (by FFMPEG API)

    memcpy(dataBuffer,packet.data,packet.size);    //directly copy the packets to buffer
    memcpy(&dataBuffer_pct[packet.size], s_frameAudInfo,VIDEO_FRAME_AUD_LEN); //add 6-byte marker
    then I just send it through DjiPayloadCamera_SendVideoStream.
     
    I try to keep the settings of the encoded video(= output.mp4) close to that of the original file(media_file/PSDK0004_ORG.mp4).
     
    $ffprobe -i output.mp4
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 640x480, 1203 kb/s, 24.08 fps, 23.98 tbr, 24k tbn, 47.95 tbc (default)
     
    And key frames of the video are present at 0.3 second intervals.
     
    ## Photo
    ↓ failure with black screen

    ↓ we can send it if I save the encoded packets to mp4.

    0
    Comment actions Permalink
  • megumu.tsukamoto

    coding miss is found!

    Though it has 2 sec latency, but it's work, thank you.

     

    0
    Comment actions Permalink
  • nmai

    Try the following settings for encoding in ffmpeg API:

            ret = av_opt_set(avContext->priv_data, "preset", "ultrafast", 0);
            ret = av_opt_set(avContext->priv_data, "tune", "zerolatency", 0);
            ret = av_opt_set(avContext->priv_data, "crf", "20", 0); // Range is 0 to 51, smaller is faster I believe.
     
    Also the bandwidth seems very limited especially when on air.  So use small dimension for your frames, for example 600x400 or something like that.
     
    1
    Comment actions Permalink
  • nmai

    Also you may want to increase the frame rate to at least 5 per second.  IIRC, 2 won't work.  3-4 is not reliable.  

    1
    Comment actions Permalink
  • megumu.tsukamoto

    Thank you so much nmai! I have done the same approach and setting ultrafast and zero latency works for me.

    1
    Comment actions Permalink
  • Josh

    Could anyone share some code samples? I am trying to do something very similar (streaming from /dev/video0 to the controller) with my Matrice 350. I have tried using some of the code from camera_emu, but it gives me an error saying it is not supported by the aircraft.

    0
    Comment actions Permalink
  • Nguyen Anh Khoa Tran

    Matrice 350 only supported by PSDK 3.5.0 or above. You can find it in github and try.

    0
    Comment actions Permalink
  • sam.schofield

    FYI (in case anyone else trying to stream video finds this forum post like I did) I believe Josh put together a super helpful document and repo detailing how to stream live video from a companion computer to the DJI controller using the PSDK.

    0
    Comment actions Permalink
  • CloboticsOperators

    @sam.schofield great work

    0
    Comment actions Permalink

Please sign in to leave a comment.

Didn't find what you were looking for?

New post