DEV Community: Sylar

TRAA MCP Server: A Powerful Screen Capture MCP Server

Sylar — Thu, 01 May 2025 14:20:15 +0000

Project repository traa-mcp, Your stars will be my greatest motivation for further improvements.

Project Introduction

TRAA MCP Server is a screen capture server based on the Model Context Protocol (MCP) and traa. It provides system screen and window enumeration capabilities, as well as screenshot capture functionality. Through the MCP protocol, the server can communicate with clients, supporting both standard input/output (stdio) and Server-Sent Events (SSE) transmission methods.

Usage in Claude Desktop

Core Features

1. Screen and Window Source Enumeration

sources = enum_screen_sources()
for source in sources:
    print(f"ID: {source.id}")
    print(f"Title: {source.title}")
    print(f"Type: {'Window' if source.is_window else 'Display'}")
    print(f"Position: {source.rect}")  # (left, top, right, bottom)

This feature can:

List all available displays in the system
List all available windows in the system
Provide detailed information for each source, including ID, title, type, and position information

2. Screen Capture

The project offers two screenshot methods:

2.1 Direct Image Data Retrieval

image = create_snapshot(
    source_id=1,           # Screen source ID
    size=(1920, 1080)      # Desired size
)

This method returns an MCP Image object, suitable for scenarios requiring direct image data processing.

2.2 Save to File

save_snapshot(
    source_id=1,           # Screen source ID
    size=(1920, 1080),     # Desired size
    file_path="screenshot.jpeg",  # Save path
    quality=80,            # Image quality (1-100)
    format="jpeg"          # Image format
)

This method directly saves the screenshot as a file, supporting:

Automatic directory creation
Configurable image quality
Support for JPEG and PNG formats

Technical Implementation

1. Architecture Design

The project adopts a modular design, mainly containing the following components:

enum_screen_sources: Enumerate screen sources
create_snapshot: Create screenshots
save_snapshot: Save screenshots
_create_snapshot: Internal implementation function

2. Error Handling

The project implements a comprehensive error handling mechanism:

try:
    result = traa.create_snapshot(source_id, size)
except ValueError as e:
    # Parameter validation error
    raise e
except traa.Error as e:
    # TRAA-related error
    raise RuntimeError(f"Failed to create snapshot: {e}")
except Exception as e:
    # Other unexpected errors
    raise RuntimeError(f"Unexpected error: {e}")

3. Image Processing

Support for RGBA and RGB format conversion
Automatic image size optimization
Configurable image quality settings

Usage Guide

1. Install Dependencies

pip install mcp>=1.0.0 anyio>=4.5 traa>=0.1.5 pillow>=11.1.0

Recommended to use uv for dependency installation:

uv sync

2. Start the Server

Supports two running modes:

stdio mode (default):

uv run traa_mcp_server

SSE mode:

uv run traa_mcp_server-sse --port 3001

3. Use the Client

uv run traa_mcp_client

4. Complete Example

# 1. Enumerate all screen sources
sources = enum_screen_sources()
for source in sources:
    print(f"Found screen source: {source.title}")

# 2. Capture screenshot of the first screen source
if sources:
    # Get image data
    image = create_snapshot(sources[0].id, (1920, 1080))

    # Save as file
    save_snapshot(
        sources[0].id,
        (1920, 1080),
        "screenshot.jpeg",
        quality=80,
        format="jpeg"
    )

Best Practices

Image Format Selection
- JPEG: Suitable for screenshots with many colors, smaller file size
- PNG: Suitable for screenshots with text or requiring transparency, better quality
Performance Optimization
- Default to JPEG format with quality set to 60 to keep file size under 1MB
- Adjust quality settings as needed
Error Handling
- Always check return values
- Use try-except to handle potential exceptions
- Review detailed error messages for troubleshooting

Summary

TRAA MCP Server provides a powerful and flexible screen capture solution. Its modular design, comprehensive error handling, and rich features make it an ideal choice for various screen capture needs. Whether you need simple screenshot functionality or complex image processing, this project can meet your requirements.

References

Flutter PIP Plugin - Android Support Auto Enter PiP Mode below Android 12

Sylar — Thu, 01 May 2025 14:03:37 +0000

Project repository PIP, also published on pub.dev pip 0.0.3. Your stars and likes will be my greatest motivation for further improvements.

According to the development documentation Add videos using picture-in-picture (PiP), the PIP feature was introduced in Android 8.0 (API level 26), but the autoEnter functionality was only supported starting from Android 12. For versions that don't support it, we need to actively call enterPictureInPictureMode by listening to onUserLeaveHint to enter PIP Mode. In previous versions, because FlutterActivity didn't forward onUserLeaveHint, we could only use flutter's didChangeAppLifecycleState event in dart to actively call the method when the app went to the background to enter PIP Mode. However, in practice, it seems impossible to distinguish notifications from the notification bar, which caused the app to automatically enter PIP Mode even when it was in the foreground when users pulled down the notification bar. This was obviously not what we wanted. The optimized effect is as follows:

Modifying the PIP Plugin

Adding PipActivity

  package org.opentraa.pip;

  import android.app.PictureInPictureUiState;
  import android.content.res.Configuration;
  import android.os.Build;
  import androidx.annotation.NonNull;
  import androidx.annotation.RequiresApi;
  import io.flutter.embedding.android.FlutterActivity;

  @RequiresApi(Build.VERSION_CODES.O)
  public class PipActivity extends FlutterActivity {
    public interface PipActivityListener {
      void onPictureInPictureModeChanged(boolean isInPictureInPictureMode,
                                        Configuration newConfig);

      void onPictureInPictureUiStateChanged(PictureInPictureUiState state);

      boolean onPictureInPictureRequested();

      void onUserLeaveHint();
    }

    private PipActivityListener mListener;

    public void setPipActivityListener(PipActivityListener listener) {
      mListener = listener;
    }

    // only available in API level 26 and above
    @RequiresApi(26)
    @Override
    public void onPictureInPictureModeChanged(boolean isInPictureInPictureMode,
                                              Configuration newConfig) {
      super.onPictureInPictureModeChanged(isInPictureInPictureMode, newConfig);
      if (mListener != null) {
        mListener.onPictureInPictureModeChanged(isInPictureInPictureMode,
                                                newConfig);
      }
    }

    // only available in API level 30 and above
    @RequiresApi(30)
    @Override
    public boolean onPictureInPictureRequested() {
      if (mListener != null) {
        return mListener.onPictureInPictureRequested();
      }
      return super.onPictureInPictureRequested();
    }

    // only available in API level 31 and above
    @RequiresApi(31)
    @Override
    public void
    onPictureInPictureUiStateChanged(@NonNull PictureInPictureUiState state) {
      super.onPictureInPictureUiStateChanged(state);
      if (mListener != null) {
        mListener.onPictureInPictureUiStateChanged(state);
      }
    }

    @Override
    public void onUserLeaveHint() {
      super.onUserLeaveHint();
      if (mListener != null) {
        mListener.onUserLeaveHint();
      }
    }
  }

The main idea is that if users of the PIP plugin want to support automatic entry into PIP Mode when the app goes to the background on Android 12 and below, they can modify their MainActivity's parent class to PipActivity. This way, when the PIP plugin is registered, it can determine whether to enable related functionality by checking if the passed Activity is a PipActivity.

Binding Activity to PipController PipPlugin initializes PipController in onAttachedToActivity and onReattachedToActivityForConfigChanges

  private void initPipController(@NonNull ActivityPluginBinding binding) {
    if (pipController == null) {
      pipController = new PipController(
          binding.getActivity(), new PipController.PipStateChangedListener() {
            @Override
            public void onPipStateChangedListener(
                PipController.PipState state) {
              // put state into a json object
              channel.invokeMethod("stateChanged",
                                   new HashMap<String, Object>() {
                                     { put("state", state.getValue()); }
                                   });
            }
          });
    } else {
      pipController.attachToActivity(binding.getActivity());
    }
  }

  @Override
  public void onAttachedToActivity(@NonNull ActivityPluginBinding binding) {
    initPipController(binding);
  }

  @Override
  public void onReattachedToActivityForConfigChanges(
      @NonNull ActivityPluginBinding binding) {
    initPipController(binding);
  }

In the PipController constructor and attachToActivity method, check if autoEnter is supported based on the current system version and bound Activity

  public PipController(@NonNull Activity activity,
                       @Nullable PipStateChangedListener listener) {
    setActivity(activity);
    //  ... Other code ...
  }

  private boolean checkAutoEnterSupport() {
    // Android 12 and above support to set auto enter enabled directly
    if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.S) {
      return true;
    }

    // For android 11 and below, we need to check if the activity is kind of
    // PipActivity since we can enter pip mode when the onUserLeaveHint is
    // called to enter pip mode as a workaround
    Activity activity = mActivity.get();
    return activity instanceof PipActivity;
  }

  private void setActivity(Activity activity) {
    mActivity = new WeakReference<>(activity);
    if (activity instanceof PipActivity) {
      ((PipActivity)activity).setPipActivityListener(this);
    }

    mIsSupported = checkPipSupport();
    mIsAutoEnterSupported = checkAutoEnterSupport();
  }

  public void attachToActivity(@NonNull Activity activity) {
    setActivity(activity);
  }

Modifying MainActivity in the Example Project

Simple MainActivity

  package org.opentraa.pip_example;

  import io.flutter.embedding.android.FlutterActivity;
  import org.opentraa.pip.PipActivity;

  public class MainActivity extends PipActivity {
  }

As shown above, we have now supported PIP Mode autoEnter functionality for all versions.

iOS Picture in Picture (PiP) Implementation Guide

Sylar — Thu, 17 Apr 2025 15:37:51 +0000

Introduction

Picture-in-Picture (PiP) is a feature that allows users to continue watching video content while using other applications. This guide provides a comprehensive walkthrough of implementing PiP functionality in iOS applications, including custom content rendering and system control management.

Demo

Features

Implemented Features

✅ Core PiP interface implementation (setup, start, stop, release)
✅ Custom content rendering with plugin separation
✅ PiP window control style management
✅ Automatic background PiP mode activation
✅ PiP window size and aspect ratio adjustment
✅ Custom content rendering demo (UIView image loop)

Planned Features

⏳ Playback event monitoring and resource optimization
⏳ Automatic implementation switching based on system version and app type
⏳ MPNowPlayingSession integration for playback information
⏳ Performance optimization and best practices

Implementation Overview

While Apple's official documentation primarily covers AVPlayer-based PiP implementation and VOIP PiP, it lacks detailed information about advanced features like custom rendering and control styles. This guide provides a complete implementation solution based on practical experience.

Core Concepts

PiP Window Display

The core implementation involves inserting a UIView (AVSampleBufferDisplayLayer) into the specified contentSourceView and rendering a transparent image. This approach enables PiP functionality without affecting the original content.
Custom Content Rendering

Instead of using the standard video frame display method, we implement custom content rendering by dynamically adding a UIView to the PiP window. This approach offers greater flexibility and better encapsulation.

Technical Considerations

Key Implementation Notes

Audio Session Configuration

Even for videos without audio, setting the audio session to movie playback is essential. Without this configuration, the PiP window won't open when the app moves to the background.
Control Management

While requiresLinearPlayback controls fast-forward/rewind buttons, other controls (play/pause buttons, progress bar) require KVO-based controlStyle configuration.
ViewController Access
Direct access to the PiP window's ViewController is not available. Two current implementation approaches:
- Add views to the current active window
- Access the Controller's private viewController property through reflection
Warning: Using private APIs may affect App Store approval. Consider seeking more stable alternatives.

Implementation Steps

1. Create PipView

PipView.h

#import <UIKit/UIKit.h>

NS_ASSUME_NONNULL_BEGIN

@class AVSampleBufferDisplayLayer;

@interface PipView : UIView

@property (nonatomic) AVSampleBufferDisplayLayer *sampleBufferDisplayLayer;

- (void)updateFrameSize:(CGSize)frameSize;

@end

NS_ASSUME_NONNULL_END

PipView.m

#import "PipView.h"
#import <AVFoundation/AVFoundation.h>

@implementation PipView

+ (Class)layerClass {
    return [AVSampleBufferDisplayLayer class];
}

- (AVSampleBufferDisplayLayer *)sampleBufferDisplayLayer {
    return (AVSampleBufferDisplayLayer *)self.layer;
}

- (instancetype)init {
    self = [super init];
    if (self) {
        self.alpha = 0;
    }
    return self;
}

- (void)updateFrameSize:(CGSize)frameSize {
    CMTimebaseRef timebase;
    CMTimebaseCreateWithSourceClock(nil, CMClockGetHostTimeClock(), &timebase);
    CMTimebaseSetTime(timebase, kCMTimeZero);
    CMTimebaseSetRate(timebase, 1);
    self.sampleBufferDisplayLayer.controlTimebase = timebase;
    if (timebase) {
        CFRelease(timebase);
    }

    CMSampleBufferRef sampleBuffer =
        [self makeSampleBufferWithFrameSize:frameSize];
    if (sampleBuffer) {
        [self.sampleBufferDisplayLayer enqueueSampleBuffer:sampleBuffer];
        CFRelease(sampleBuffer);
    }
}

- (CMSampleBufferRef)makeSampleBufferWithFrameSize:(CGSize)frameSize {
    size_t width = (size_t)frameSize.width;
    size_t height = (size_t)frameSize.height;

    const int pixel = 0xFF000000; // {0x00, 0x00, 0x00, 0xFF};//BGRA

    CVPixelBufferRef pixelBuffer = NULL;
    CVPixelBufferCreate(NULL, width, height, kCVPixelFormatType_32BGRA,
                        (__bridge CFDictionaryRef)
                            @{(id)kCVPixelBufferIOSurfacePropertiesKey : @{}},
                        &pixelBuffer);
    CVPixelBufferLockBaseAddress(pixelBuffer, 0);
    int *bytes = CVPixelBufferGetBaseAddress(pixelBuffer);
    for (NSUInteger i = 0, length = height *
                                    CVPixelBufferGetBytesPerRow(pixelBuffer) / 4;
         i < length; ++i) {
        bytes[i] = pixel;
    }
    CVPixelBufferUnlockBaseAddress(pixelBuffer, 0);
    CMSampleBufferRef sampleBuffer =
        [self makeSampleBufferWithPixelBuffer:pixelBuffer];
    CVPixelBufferRelease(pixelBuffer);
    return sampleBuffer;
}

- (CMSampleBufferRef)makeSampleBufferWithPixelBuffer:
    (CVPixelBufferRef)pixelBuffer {
    CMSampleBufferRef sampleBuffer = NULL;
    OSStatus err = noErr;
    CMVideoFormatDescriptionRef formatDesc = NULL;
    err = CMVideoFormatDescriptionCreateForImageBuffer(kCFAllocatorDefault,
                                                       pixelBuffer, &formatDesc);

    if (err != noErr) {
        return nil;
    }

    CMSampleTimingInfo sampleTimingInfo = {
        .duration = CMTimeMakeWithSeconds(1, 600),
        .presentationTimeStamp =
            CMTimebaseGetTime(self.sampleBufferDisplayLayer.timebase),
        .decodeTimeStamp = kCMTimeInvalid};

    err = CMSampleBufferCreateReadyWithImageBuffer(
        kCFAllocatorDefault, pixelBuffer, formatDesc, &sampleTimingInfo,
        &sampleBuffer);

    if (err != noErr) {
        return nil;
    }

    CFRelease(formatDesc);

    return sampleBuffer;
}

@end

2. Configure PiP Controller

// Create PipView
PipView *pipView = [[PipView alloc] init];
pipView.translatesAutoresizingMaskIntoConstraints = NO;

// Add to source view
[currentVideoSourceView insertSubview:pipView atIndex:0];
[pipView updateFrameSize:CGSizeMake(100, 100)];

// Create content source
AVPictureInPictureControllerContentSource *contentSource =
    [[AVPictureInPictureControllerContentSource alloc]
        initWithSampleBufferDisplayLayer:pipView.sampleBufferDisplayLayer
                        playbackDelegate:self];

// Create PiP controller
AVPictureInPictureController *pipController =
    [[AVPictureInPictureController alloc] initWithContentSource:contentSource];
pipController.delegate = self;
pipController.canStartPictureInPictureAutomaticallyFromInline = YES;

3. Configure Control Style

// Control fast-forward/rewind buttons
pipController.requiresLinearPlayback = YES;

// Control other UI elements
[pipController setValue:@(1) forKey:@"controlsStyle"]; // Hide forward/backward, play/pause buttons and progress bar
// [pipController setValue:@(2) forKey:@"controlsStyle"]; // Hide all system controls

4. Implement Playback Delegate

- (CMTimeRange)pictureInPictureControllerTimeRangeForPlayback:
    (AVPictureInPictureController *)pictureInPictureController {
    return CMTimeRangeMake(kCMTimeZero, kCMTimePositiveInfinity);
}

5. Manage Custom View

// Add custom view
- (void)pictureInPictureControllerDidStartPictureInPicture:
    (AVPictureInPictureController *)pictureInPictureController {
    [pipViewController.view insertSubview:contentView atIndex:0];
    [pipViewController.view bringSubviewToFront:contentView];

    // Configure constraints
    contentView.translatesAutoresizingMaskIntoConstraints = NO;
    [pipViewController.view addConstraints:@[
        [contentView.leadingAnchor constraintEqualToAnchor:pipViewController.view.leadingAnchor],
        [contentView.trailingAnchor constraintEqualToAnchor:pipViewController.view.trailingAnchor],
        [contentView.topAnchor constraintEqualToAnchor:pipViewController.view.topAnchor],
        [contentView.bottomAnchor constraintEqualToAnchor:pipViewController.view.bottomAnchor],
    ]];
}

// Remove custom view
- (void)pictureInPictureControllerDidStopPictureInPicture:
    (AVPictureInPictureController *)pictureInPictureController {
    [contentView removeFromSuperview];
}

References

Project Links

Your support is appreciated!