But the next problem is obvious: How can you logon securely from your laptop to those isolated VMs, which are not in your home network, without opening ports to the public internet?

In this guide, I'll explain how to connect your Tailscale mesh network to a Proxmox SDN virtual network with Subnet Routing so that you can log into any VM on your private SDN easily via SSH.

The Architecture Overview

Rather than having to install the Tailscale client on each individual virtual machine, we can make our Proxmox host (or a gateway container/VM on it) a Tailscale Subnet Router.

Tailscale Network (Tailnet): Your secure overlay network, over all your personal devices.

Simple SDN Zone: Your isolated internal network (e.g., 10.10.10.0/24) in which your VMs reside.

The Bridge: Tailscale runs on the Proxmox host, which advertises the SDN subnet route, which means that traffic securely passes through the encrypted tunnel directly into your internal VMs.

The Process

The first step is to advertise the SDN Subnet on the Proxmox Host.

Log in your remote Proxmox host using SSH. If you've already installed Tailscale on the host, you must inform Tailscale to advertise the internal IP range of your SDN to your tailnet.

Execute the following command (substitute 10.10.10.0/24 with the range of your Proxmox Simple SDN network):

sudo tailscale up --advertise-routes=10.10.10.0/24

As a note, if your Tailscale daemon was already running, you can use the command tailscale set --advertise-routes=10.10.10.0/24 instead.

Next, you need to enable the IP Forwarding function in the Linux kernel.

By default, Linux will drop packets received from other networks as foreign. Your Proxmox host must be configured as a router between Tailscale (tailscale0) and your SDN bridge, so you need to set up IPv4 and IPv6 packet forwarding.

1. Make a sysctl config file for Tailsale:

echo 'net.ipv4.ip_forward = 1' | sudo tee -a /etc/sysctl.d/99-tailscale.conf
echo 'net.ipv6.conf.all.forwarding = 1' | sudo tee -a /etc/sysctl.d/99-tailscale.conf

1. Apply the changes right away:

sudo sysctl -p /etc/sysctl.d/99-tailscale.conf

Then, In the Tailscale admin console, approve the route.

For security reasons, Tailscale doesn't automatically trust nodes that start routing traffic for external subnets. You have to explicitly approve it:

Go to Tailscale Admin Console.

Find your Proxmox host in the list. If you're using a mobile device, click on the three-dot menu on the right and then on “Edit route settings.”

Check the box to Approve the advertised subnet route (10.10.10.0/24).

Finally, tell your local machine to accept routes:

For Tailscale to be able to propagate custom routes advertised by your tailnet nodes, it requires you to tell it that this is OK from your local client machine (the laptop you are connecting from).

Run this on your local machine:

sudo tailscale set --accept-routes=true

Once you have finished, you can verify your work and SSH.

The tunnel is now in use. You should now be able to ping or SSH directly into any VMs that you have in your Proxmox SDN, using their local IP address, from your local machine.

ssh username@10.10.10.55

Pro-Tip: Firewall Checks

If your connection hangs or times out, you can check that UFW or your Proxmox host's firewall are not aggressively blocking forwarding rules from the tailscale0 interface. Consider temporarily allowing traffic from Tailsace with:

sudo ufw allow in on tailscale0

Conclusion

And that's it! You now have secure, encrypted access to your virtual machines running in your isolated Proxmox SDN from anywhere in the world, developer-ready without having to punch a hole in your firewall or have to put in a complex port-forwarding rule on your home router.

Detailed Observations

Multiple Projects & Permission Management

One of the first things that stood out to me was Langfuse's sophisticated and granular permission management system. It's incredibly well-designed for environments needing precise control. I found it supports creating multiple projects, each with its own distinct roles and permissions. The availability of both organization-level and project-level roles allows for incredibly fine-tuned control over user access and actions. For instance, an "Owner" at the organization level has extensive permissions, covering everything from project creation and organization management (like API keys, updates, deletion, member management, and billing) to specific project actions. Within individual projects, the roles are even more detailed, covering comprehensive control over functionalities like:

• Project Management: The ability to read, update, and delete projects, manage members, API keys, and integrations.

• Data & Observability: Permissions to publish, bookmark, tag objects, delete traces, manage scores and configurations, and handle datasets, prompts, models, and evaluation templates.

• LLM Integration: Control over LLM API keys, schemas, and tools.

• Data Export & Automation: Managing batch exports, comments, annotation queues, prompt experiments, audit logs, dashboards, and automations.

This level of control is genuinely impressive and crucial for larger teams or projects with varying access needs.

Key Features I Noticed

Several features really caught my attention during testing:

1. LLM Playground: The dedicated LLM Playground is a great addition, making it easy to experiment with different large language models.

2. LLM Model Connections & Pricing: Langfuse offers extensive out-of-the-box connections to a wide array of LLM models, and crucially, includes their associated price details. This is incredibly valuable for cost tracking and optimization, something often overlooked in other tools.

3. Slack Notification Support: The platform supports Slack notifications, which is fantastic for real-time alerts and keeping teams updated on critical events.

4. Trace Data Export: The ability to export trace data to S3-compatible external storages, with the option to schedule these exports, is a practical feature for data retention and analysis.

5. Dashboards:

   • Prebuilt Dashboards: Langfuse provides useful prebuilt dashboards for cost, usage, and latency. However, I did notice that community dashboards seem limited at this point.

   • Custom Dashboards: While custom dashboards and widgets are supported, a notable limitation I found was the absence of an option to export or share these dashboards across different projects.

6. Prompt Engineering & Management: This is a significant strength of Langfuse.

   • It offers a centralized repository for managing prompts, which is a huge advantage for organization.

   • Users can organize prompts into folders, which is helpful for larger prompt libraries.

   • Version control for prompts is supported, allowing for tracking changes and easy rollbacks – a critical feature for iterative prompt development.

   • Prompts can be called directly within a project using the Langfuse client, which really streamlines integrating managed prompts into applications.

   • The support for webhook calls and automations on any CRUD changes to a prompt enables dynamic responses to prompt modifications, which opens up interesting possibilities.

7. Tracing & Metrics:

   • Traces: Langfuse's tracing capabilities are robust, including token and cost tracking. The rate of trace collection can also be throttled using a sampling value.

   • Sensitive Information Masking: The platform allows for masking sensitive information before it's sent to the server. This is achieved by writing a masking function that matches sensitive information by pattern and applying it to the Langfuse client – a crucial security feature.

   • Observations, Sessions, and Users Tracking: Comprehensive tracking of observations, sessions, and users is well-facilitated.

   • SDK Support: My testing primarily used the Python SDK, but it has support for other languages, including JavaScript, Langchain Python, and Langchain JS.

   • UI Features: The UI offers various features for analyzing trace data, including span and timeline views, filtering, searching, data download, cost breakdown, and temperature details – making analysis quite user-friendly.

8. Playground: The playground has strong out-of-the-box LLM connections for major providers like Google Vertex AI, Google AI Studio, Anthropic, OpenAI, Azure, and Bedrock. It also supports other LLMs through custom API calls, offering flexibility.

9. Asynchronous Batch Data Collection: Langfuse is designed to send trace data in batches asynchronously.

10. Security Monitoring: I noted that the platform includes capabilities for security monitoring, specifically collecting traces about guardrails like llm_guards and their effectiveness in blocking banned topics.

11. MCP Server Integration: The inclusion of an MCP server simplifies integrating langfuse tracing in an application. MCB is available for various AI coding agents like Cursor and Copilot.

12. Evaluation: The evaluation framework is a definite standout.

    • LLM as Evaluator: The ability to leverage LLMs themselves to act as evaluators is a powerful feature for automated assessment.

    • Scoring Categories & Manual Assignment: The platform allows for setting up scoring configs and manually assigning scores to observations.

    • Dataset-Based Evaluation: Running evaluations against predefined datasets of questions and expected outputs provides a structured and repeatable approach to assessing model performance.

Pricing

Langfuse offers a sensible tiered pricing model, which I think caters well to various user needs. The most appealing aspect for many will be the self-hosted (open-source) option, which provides all core features (observability, evaluation, prompt management, and datasets) for free. This gives users complete control over their data and infrastructure. Langfuse Cloud is also available for those who prefer a managed service.

Conclusion

To sum it up, Langfuse is, in my observation, a highly effective and specialized tool for AI model observability. Its dedicated features for AI, particularly its robust scoring, evaluation, and prompt management capabilities, provide significant advantages over more general-purpose observability tools. While its use of OpenTelemetry means some functional overlap exists, Langfuse's focused approach gives it a distinct edge in the AI domain. The strong permission management, extensive LLM integrations, and comprehensive tracing and evaluation features make it a truly valuable asset for anyone working with AI models. I found its strengths in prompt management and its advanced evaluation features to be particularly handy.

Example Using Annotations

from google.genai import Client, types
from langfuse import Langfuse, observe
from dotenv import load_dotenv
from openinference.instrumentation.google_genai import GoogleGenAIInstrumentor

load_dotenv()

client = Client(api_key="")
langfuse = Langfuse()
GoogleGenAIInstrumentor().instrument()


@observe(as_type="generation")
def generate():
    langfuse.update_current_trace(user_id="bishal")
    prompt = langfuse.get_prompt("glaze/female", label="production")
    langfuse.update_current_generation(
        prompt=prompt,
    )
    compined_prompt = prompt.compile(name="")
    response = client.models.generate_content(
        model="gemini-2.5-flash",
        config=types.GenerateContentConfig(
            temperature=0.2,

        ),
        contents=compined_prompt,
    )
    print(response.text)


generate()

Screenshots

The Incident

It all started during what should have been routine virtual machine management at my client's site. Without anyone deleting anything on purpose, a system anomaly led to the inadvertent removal of critical files from an office server VM. This immediately caused problems with their services, disrupted daily internal operations, and raised significant concerns about their data integrity.

The Recovery Process

My recovery plan for the client was clear: to restore the entire VM's functionality, including its configuration, binary files, and any application data like MySQL tables. Just copying files wasn't an option, as crucial setup information was missing. I had to approach this with precision and care.

Getting Ready for Recovery

1. Initial Assessment and Containment: As soon as I was brought in, my first priority was to quickly assess the full extent of the data loss and immediately contain the incident. This was crucial to prevent any further data corruption or potential spread of the issue.

2. Set Up the Recovery Environment: I established a safe, isolated recovery environment at the client's location. First, I took a backup of what was left on the affected system. This involved securing all their existing backups too, and making sure I had the right permissions to do the work without affecting their live systems.

1. Attempting Direct Disk Recovery and VM Reconstruction

This phase involved my initial, fastest attempt at recovery directly from the affected disk, followed by VM reconstruction if needed.

1. Attempting Direct File Recovery with TestDisk (My First Line of Defense):

   My client's last full VM backup was 15 days old, meaning a direct restore would lead to a significant data loss gap. To minimize this, I first attempted to recover files directly from the deleted disk. This was my fastest path to potentially recovering the most recent data.

   The reason this approach holds promise is due to how file deletion typically works in Linux systems. When a file is "deleted" on a Linux filesystem, the data itself isn't immediately erased from the disk sectors. Instead, the operating system primarily removes the file's metadata (like its name, location, and size) from the filesystem's index. The sectors where the data resides are merely marked as available for new data. Until new data overwrites those sectors, the original data might still be present.

   I utilized TestDisk, a powerful free and open-source data recovery utility, designed to recover lost partitions and make non-booting disks bootable again, but it's also highly effective at recovering deleted files.

   Here's a simplified overview of how I used TestDisk:

   • Installation: TestDisk was installed on a separate live Linux environment.

       sudo apt-get install testdisk
     

   • Disk Selection: I launched testdisk and selected the affected disk drive where the VM data was lost.

   • Partition Analysis: TestDisk was instructed to analyze the partition structure, looking for lost partitions or deleted files.

   • File Recovery: I navigated through the detected filesystem structure, searching for the critical VM configuration files and binary files that were reported missing. TestDisk attempts to scan the unallocated sectors and rebuild the file metadata, allowing for recovery.

Outcome of TestDisk:

Unfortunately, despite my best efforts and TestDisk's capabilities, in this specific instance, the tool did not yield the desired results for the critical VM configuration and and binary files. This suggested that the sectors containing the crucial metadata or parts of the files might have already been overwritten or were too fragmented for a complete recovery in this complex VM environment. While this was a setback, it was a necessary and rapid first attempt to minimize data loss.

2. Identifying Post-Backup Changes and Data Location

After TestDisk didn't provide a full recovery, I shifted focus to identifying what data might have changed since the 15-day-old backup. I determined that:

• Most of the core system configuration files would likely be the same as in the backup.

• The Gitea repository (for code) and its commit data would definitely have changed.

• The MySQL database data would also have been updated.

Luckily, I found that the /var/lib directory on the corrupt disk still contained the data for both MySQL and Gitea. This was a crucial discovery, as these directories typically store the application's persistent data.

3. VM Reconstruction and Configuration Restoration:

Since a direct recovery wasn't fully successful, I proceeded to recreate the VM environment.

• New VM Setup: I set up a brand new Ubuntu 22 VM (matching the client's original operating system) to serve as the recovery target.

• Package Installation: All necessary software packages were installed on this new VM.

• Configuration Replacement: I then replaced the default configuration files on the new VM with the configuration files recovered from the 15-day-old backup. This brought the system's core settings back to a known good state.

Firewall Rules and Gitea Data Recovery:

• Firewall Rules: The client's firewall rules were well-documented in their change logs. I manually recreated these rules on the new VM, a process that did not take much time.

• Gitea Data Recovery: For the Gitea application, I simply replaced its data directory on the new VM with the Gitea data directory I had recovered from the corrupt disk's /var/lib location. After carefully managing file permissions and ownership, database recovery was the only thing left to bring Gitea and other services online with its latest data.

4. MySQL Data Recovery – The Core Challenge

After the configurations were restored, the most important task was recovering the MySQL data. This required a detailed approach, especially since the .ibd files were crucial and directly copying them wasn't straightforward due to missing metadata.

1. Database Structure Re-initialization:

   The foundational step was to re-establish the database's structural integrity. I did this by importing an SQL dump from the older, 15-day-old backup. This recreated all database schemas, table definitions, and initial configurations.

    sudo mysql < backup.sql
   

2. User Privilege Configuration:

   Following the schema restoration, I recreated the necessary database user accounts for the client and reinstated their respective privileges. All user credentials were securely retrieved from backed-up configurations files.

3. Understanding InnoDB and .ibd File Restoration:

   To effectively restore the client's MySQL database, I leveraged my basic understanding of the MySQL InnoDB storage engine, particularly its transactional properties and recovery mechanisms. This was crucial for handling the .ibd files.

   • Temporarily Disable Database Rules:

     To ensure a smooth import process and prevent any integrity errors, I temporarily turned off foreign key checks.

       SET FOREIGN_KEY_CHECKS = 0;
     

   • Discard Existing Data Links (Tablespaces): To prepare for the seamless import of the new .ibd files, I first told the database to forget where its current data files were. This effectively unlinks the table definitions from their associated data files, clearing the way for the new ones.

     Important Note: This operation disconnects the current data. I always ensure my clients have a complete and verified backup before performing this step!

       SELECT CONCAT('ALTER TABLE `', TABLE_SCHEMA, '`.`', TABLE_NAME, '` DISCARD TABLESPACE;') AS sql_command FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA = <DatabaseName> AND ENGINE = 'InnoDB';
     

     I executed each command generated by this query individually to maintain precise control.

   • Copy and Secure the Data Files:

     After discarding the old links, I carefully copied recovered .ibd files from the corrupt disk into the corresponding database folder on the new VM. It was crucial to set the right ownership for these files. Incorrect permissions would have prevented MySQL from accessing them.

       /old/var/lib/mysql/<DatabaseName>/ # Source path on the corrupt disk
       chown mysql:mysql /var/lib/mysql/<DatabaseName>/*
     

   • Import the New Data (Tablespaces):

     With the .ibd files correctly positioned and permissions accurately configured, the next step was to instruct MySQL to use these new data files. This action successfully linked the table designs with their actual data, bringing them back online.

       SELECT CONCAT('ALTER TABLE `', TABLE_SCHEMA, '`.`', TABLE_NAME, '` IMPORT TABLESPACE;') AS sql_command FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA = <DatabaseName> AND ENGINE = 'InnoDB';
     

     I executed the resulting commands sequentially for each table.

   • Re-enable Database Rules: Upon the successful import of all data, I promptly turned the foreign key checks back on. This restored the rules that maintain data consistency and integrity within the database for all future operations.

       SET FOREIGN_KEY_CHECKS = 1;
     

5. Addressing Advanced Recovery Challenges (Schema Mismatches):

A significant hurdle emerged when I discovered that 19 out of 127 tables of "Gitea" had different schema. This was primarily due to recent updates to the Gitea application, which prevented a straightforward .ibd file restoration for these specific tables.

Leveraging ibd2sql for Data Extraction:

For tables with schema discrepancies or where standard recovery methods were ineffective, I employed a specialized tool called ibd2sql. This utility is designed to parse InnoDB .ibd files and extract the contained data as SQL INSERT statements, even when a complete MySQL instance or .frm files are unavailable.

cd ibd2sql
for path in $(realpath /old/var/lib/mysql/<DatabaseName>); do python3 main.py $path --sql --ddl | mysql; done

Important Note: While ibd2sql is a powerful recovery tool, it may not fully support all complex data types, and I always recommend thorough post-recovery data validation.

In instances where ibd2sql provided table designs that didn't perfectly align with the current schema, manual intervention was required. I meticulously consulted the client's application's (Gitea's) official database schema changelogs and migration scripts. These resources provided the precise Data Definition Language (DDL) for each table at various points in time, allowing me to accurately reconstruct the correct table definitions.

After carefully modifying the table designs to reflect the correct schema gleaned from the changelogs, I tested them. This involved creating temporary tables and performing small-scale data imports to ensure the design precisely matched the .ibd file structure and that data integrity was fully maintained. This meticulous manual process, combined with leveraging historical schema information, was crucial for successfully recovering data from files that initially seemed lost.

6. Validation and Final Checks

1. System Integrity Checks: After all files were restored, we performed comprehensive checks to ensure the VM and all its applications were running correctly and without errors. This included verifying file integrity and system services.

2. Data Validation: For critical applications like the database and Gitea, I worked closely with the client to validate the restored data, ensuring its accuracy and completeness.

In Conclusion

The VM data loss incident at my client's site was undoubtedly a demanding experience, but it also proved to be a profound learning opportunity for both them and me. It underscored the critical importance of proactive prevention, meticulous planning, and the ability to maintain composure and execute methodically under pressure. Getting their data back successfully has strengthened their reliance on robust data management practices, and I am proud to have been their partner in this recovery. By implementing these reinforced best practices, I am confident that I can help my clients turn potential data catastrophes into manageable incidents, ensuring their services run smoothly and their data remains safe.

My Hardware:

• CPU: 8 Core, Intel(R) Core(TM) i5-9300H CPU @ 2.40GHz

• Memory: 2x 8 GB DDR4

• GPU: Nvidia GeForce GTX 1650 Mobile / Max-Q

• Proxmox Version: 8.3.2

It's a testament to the fact that you don't necessarily need cutting-edge hardware to explore the world of large language models.

In this guide, I will explain step by step how to set up Deepseek with GPU passthrough in Proxmox. A big shoutout to KasperSkytte's gist for guiding the GPU passthrough setup.

Setup Proxmox for GPU passthrough

First and foremost, we have to make some changes to our Proxmox host to enable GPU passthrough. Let’s start by updating the grub. In the /etc/default/grub file make these changes and update grub.

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb nomodeset video=vesafb:off,efifb:off"

update-grub

Next, we have to load VFIO modules. Edit /etc/modules file and add these lines:

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

The commands below enable IOMMU interrupt mapping.

echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/iommu_unsafe_interrupts.conf
echo "options kvm ignore_msrs=1" > /etc/modprobe.d/kvm.conf

We don’t want our Proxmox host to use the GPU, so let us blacklist the Nvidia drivers in the Proxmox host so that our Proxmox host does not use the GPU.

echo "blacklist radeon" >> /etc/modprobe.d/blacklist.conf
echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf
echo "blacklist nvidia" >> /etc/modprobe.d/blacklist.conf

Now, adding a GPU to VFIO can be tricky. The device IDs and vendor IDs can be confusing, and there's a high chance of making typos that could cause your setup to fail. So, be extra careful during this step.

Run this command.

lspci -v

This will output all your PCI devices. Look for the lines that show your GPU. It'll look something like this:

01:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1) (prog-if 00 [VGA controller])

01:00.1 Audio device: NVIDIA Corporation GP104 High Definition Audio Controller (rev a1)

Make note of the first set of numbers (e.g. 01:00.0 and 01:00.1). We'll need them for the next step.

Run the command below. Replace 01:00 with whatever number was next to your GPU when you ran the previous command:

lspci -n -s 01:00

Doing this should output your GPU card's Vendor IDs, usually one ID for the GPU and one ID for the Audio bus. It'll look a little something like this:

01:00.0 0000: 10de:1b81 (rev a1)

01:00.1 0000: 10de:10f0 (rev a1)

What we want to keep, are these vendor id codes: 10de:1b81 and 10de:10f0.

Now we add the GPU's vendor id's to the VFIO (remember to replace the id's with your own!):

echo "options vfio-pci ids=10de:1b81,10de:10f0 disable_vga=1"> /etc/modprobe.d/vfio.conf

Finally, run this command. This will update initramfs and reboot your Proxmox Host.

update-initramfs -u
reboot

Change VM Configuration

I had already created a Ubuntu VM in my Proxmox Host before this all. So, I just updated my VM configurations. You can create a new VM using the regular VM creation process. But there are a few things you need to configure for your VM to get direct access to the GPU.

Your VM config file, located in /etc/pve/qemu-server/<vmid>.conf should have these settings:

args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=off'
balloon: 0
bios: ovmf
cpu: host,hidden=1,flags=+pcid
hostpci0: 0000:01:00,pcie=1 # Replace with your GPU's Device ID
machine: q35

You can add the above lines anywhere in the config file and Proxmox will automatically format it in their respective place on save.

You can also change these configurations from Proxmox GUI.

Disable Secure Boot in VM

If secure boot is enabled in your BIOS, the NVIDIA kernel module might not load properly. So, make sure to disable secure boot in the VM BIOS settings. To enter the VM BIOS, press the Esc key while the Proxmox logo is showing at the start of the boot.

Setting up Ollama and Deepseek

Ollama makes it super easy to run large language models like Deepseek right on your own computer. It handles all the messy bits, so you can focus on using the model.

First, install Ollama. Just open VM console and paste this command:

curl -fsSL https://ollama.com/install.sh | sh

This command will download and install Ollama, and even take care of setting up the necessary NVIDIA drivers for your GPU.

Next, grab the Deepseek model itself:

ollama pull deepseek-r1

This downloads the Deepseek model files to your computer. You can see a list of all available models (including Deepseek) by typing:

ollama ls

Now, fire up the Deepseek model:

ollama run deepseek-r1

To check which models are currently running, and to see if your GPU is being used (which it should be ), use this command:

ollama ps

Deepseek should be now running and accessible through Ollama's API. It will be listening on port 11434 on localhost.

To talk to Deepseek using API, you can use a tool like curl. Here's an example:

curl --location 'http://localhost:11434/api/chat' \
--header 'Content-Type: application/json' \
--data '{
  "model": "deepseek-r1",
  "messages": [
    {
      "role": "user",
      "content": "Will AI replace software engineers?"
      }
  ],
  "stream": false
}'

Thank you!

By default, unattended upgrades focus on installing security updates. It usually handles updates from the -security repository. A typical configuration, split across two files, might look like this:

/etc/apt/apt.conf.d/20auto-upgrades

APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";

/etc/apt/apt.conf.d/50unattended-upgrades¹

Unattended-Upgrade::Allowed-Origins {
    "${distro_id}:${distro_codename}";
    "${distro_id}:${distro_codename}-security";
    "${distro_id}ESMApps:${distro_codename}-apps-security";
    "${distro_id}ESM:${distro_codename}-infra-security";
};
Unattended-Upgrade::Package-Blacklist {
};
Unattended-Upgrade::DevRelease "auto";

While I appreciate the security focus, I've also seen some downsides. In a corporate environment, where a dedicated team handles patching, unattended upgrades can be a bit of a headache. Imagine a critical system getting updated without the sysadmins knowing! This can lead to unexpected downtime or compatibility issues. That's why Debian itself recommends monitoring your systems closely if you use unattended upgrades. They suggest installing apt-listchanges to get email notifications about updates. You can also check the logs at /var/log/dpkg.log and /var/log/unattended-upgrades/ for details on what's been updated.

So, what if you decide unattended upgrades aren't for you? Disabling them is pretty straightforward. You can edit the unattended-upgrades configuration files and adjust the settings. For example, setting APT::Periodic::Unattended-Upgrade to "0" in /etc/apt/apt.conf.d/20auto-upgrades would disable the automatic upgrades. Alternatively, you can remove the unattended-upgrades package altogether:

Bash

sudo apt remove unattended-upgrades

Beyond just removing the package, you can also use systemctl to mask and disable the related services, providing an extra layer of prevention:

Bash

sudo systemctl mask unattended-upgrades.service
sudo systemctl disable unattended-upgrades.service

Masking prevents the service from being easily started, even manually, and disabling it prevents it from starting on boot.

If you want to customize the frequency of the unattended upgrades, you can adjust the systemd timers. More information on modifying the download and upgrade schedules can be found here: https://wiki.debian.org/UnattendedUpgrades#Modifying_download_and_upgrade_schedules_.28on_systemd.29

In my opinion, unattended upgrades feel a bit like a "Windows" approach. The Linux world, in my experience, generally prefers more control and transparency. That's probably why Debian has changed its default behavior in Debian 12 (Bookworm). It no longer installs unattended-upgrades by default with GNOME. Instead, it notifies you about available updates and lets you decide when to install them. I think this is a much better approach. People who want automatic updates can still opt-in, but it's not forced on everyone.

Ultimately, the decision of whether or not to use unattended upgrades depends on your specific needs and preferences. If you're a home user and want a hands-off approach to security, it might be a good fit. But in a corporate environment, where control and monitoring are crucial, it might be best to manage updates manually.

However, backing up VMs can sometimes be a performance-intensive task. This can lead to high CPU usage on the Proxmox VE server, as well as slow VM performance.

High CPU usage on the Proxmox VE server can be due to many factors like no of IO workers running, type of compression being used, speed of the underlying storage disks and many more.

In this blog we are going to investigate high CPU usage in the guest VMs during backups.

Cause

When a backup job for a VM is initiated, QEMU implements a "copy-before-write" mechanism in its block layer. This mechanism ensures that data required for the backup is securely transferred to the backup target before the guest VM overwrites it. To achieve this, the VM's I/O operations are temporarily blocked until the backup process catches up. If the transfer process is slow, the guest VM uses more CPU cycles waiting for I/O operations to finish, resulting in a significant rise in CPU utilization.

In simple words, During the backup transfer process VM I/O is blocked to prevent inconsistencies. If this transfer is slow, I/O operations in the VM will have to wait longer. This block on I/O operations increases the I/O wait time thus increasing the CPU usage.

If you are using Proxmox Backup Server, you can identify the bottleneck using this command.

proxmox-backup-client benchmark --repository <Your repository>

Example Output:

Uploaded 10 chunks in 17 seconds. 
Time per request: 1742814 microseconds. 
TLS speed: 2.41 MB/s 
SHA256 speed: 1831.82 MB/s 
Compression speed: 555.06 MB/s 
Decompress speed: 664.12 MB/s 
AES256/GCM speed: 1487.40 MB/s 
Verify speed: 483.77 MB/s

In the above example you can see that TLS speed is the main bottleneck. So, backups are transferring at the slow rate of 2.41 MB/s.

Backup Fleecing

Proxmox VE 8.2 introduced a significant enhancement to its backup process by introducing backup fleecing. This innovative feature aims to mitigate the performance impact of backups on guest VMs, particularly the CPU-intensive "copy-before-write" mechanism.

How Backup Fleecing Works

• Traditional Backup: In a traditional backup, QEMU ensures data consistency by transferring modified blocks to the backup target before the guest VM overwrites them. This can lead to significant I/O latency for the guest VM, resulting in increased CPU usage and potential performance degradation.

• Backup Fleecing in Action: With fleecing enabled, instead of directly sending modified blocks to the backup target, Proxmox VE temporarily caches them in a "fleecing image" on fast local storage. This local caching significantly reduces I/O latency for the guest VM, as it no longer needs to wait for slower network transfers during the initial write phase.

Benefits of Backup Fleecing:

• Improved Guest VM Performance: Reduced I/O latency translates to lower CPU usage within the guest VM, minimizing performance impacts and preventing potential freezes.

• Enhanced Backup Reliability: By minimizing I/O contention, fleecing can contribute to more stable and reliable backup operations.

Considerations for Fleecing Storage:

• Performance: Choose a fast local storage option with high I/O throughput (e.g., LVM-thin, RBD, ZFS with sparse 1).

• Thin Provisioning: Utilizing thin provisioning techniques (like LVM-thin or ZFS with sparse 1) optimizes storage utilization by only allocating space as needed for the fleecing images.

• Discard Support: If your storage supports discard operations, enable them to reclaim unused space within the fleecing images.

Enabling Backup Fleecing:

• Node-wide Configuration: Edit /etc/vzdump.conf and add:

    fleecing: enabled=true,storage=local-lvm
  

  (Replace local-lvm with the name of your chosen fleecing storage.)

• Job Specific Configuration using Proxmox VE UI: Configure fleecing in the "Advanced" tab when editing a specific backup job.

Conclusion

Backup Fleecing is a valuable feature in Proxmox VE that significantly improves the performance and reliability of backup operations by minimizing the impact on guest VM performance. By strategically implementing fleecing and selecting appropriate storage, you can ensure that your backups complete efficiently while maintaining optimal performance for your virtualized workloads.

Preparing Your Server for Proxmox Installation

Before diving into the Proxmox installation process, there are a few crucial steps to prepare your Dell server:

1. Convert to Non-RAID Disk

To ensure seamless compatibility with Proxmox, you'll need to switch your server's disks to non-RAID mode. Follow these steps:

1. Open Lifecycle Control.

2. Navigate to System Setup.

3. Access Advanced Hardware Configuration.

4. Go to Device Settings.

5. Under RAID Controller, select Configure.

6. Opt to Convert to Non-RAID disk, ensuring to check all options before confirming.

2. Switch Embedded SATA to AHCI Mode

Next up, it's time to configure the SATA settings for optimal performance with Proxmox:

1. Within Lifecycle Controller, access System Setup.

2. Navigate to System BIOS.

3. Locate SATA Settings.

4. Choose AHCI mode for the Embedded SATA from the dropdown menu.

Installation Steps

Now that your server is primed and ready, let's dive into the installation process for Proxmox:

1. Open the Virtual Console from iDRAC for remote management.

2. Download the Proxmox ISO image from the official website.

3. Create a bootable USB drive with the Proxmox image and connect it to your server.

4. Access the virtual console and boot from the USB drive.

5. Select "Install using Graphical Method" to initiate the installation wizard.

6. Accept the license agreement and proceed.

7. Choose the BOSS disk (with a 250GB NVMe SSD) and select XFS as the file system.

8. Set your country, keyboard layout, timezone, and continue.

9. Enter the hostname, Server Management IP, Gateway, DNS, and choose the Management Interface.

10. Verify your settings and continue with the installation.

11. Once completed, reboot your server to finalize the installation.

Knowledge Base

Understanding AHCI Mode

AHCI, or Advanced Host Controller Interface, is a standard interface that facilitates communication between software and SATA devices. It supports features like Hot Swapping and Native Command Queueing, optimizing performance for HDDs. While not specifically tailored for SSDs (NVMe is preferred for SSDs), AHCI remains a vital component in ensuring efficient data transfer and storage management.

NUMA and Its Significance

Non-Uniform Memory Access (NUMA) architecture plays a crucial role in modern computing environments, particularly in systems with multiple processors or cores. Understanding NUMA can greatly enhance your server's performance and resource allocation efficiency. Check out this video for a detailed explanation.

Troubleshooting: Unable to Access Proxmox from Web Browser

Encountering difficulties accessing Proxmox via your web browser? Here's a quick fix:

Solution: Ensure that the hostname specified in both /etc/hostname and /etc/hosts files matches the hostname displayed in the shell prompt (root@hostname). Consistency between these files is crucial for seamless web access.

By following these steps and understanding the underlying concepts, you can successfully install Proxmox on your Dell server, unleashing its full potential for virtualization tasks. Whether you're setting up a home lab or managing enterprise-level workloads, Proxmox on Dell servers offers a reliable and flexible solution for your virtualization needs.

Generating and Managing Keys

• Generate a key pair: Use gpg --full-generate-key to create a unique pair of keys for encryption and signing.

• List key pairs: View all your available keys with gpg --list-keys --keyid-format=long.

• Display your public key: Share your public key with others using gpg -a --export <Your Key ID>. Never share your private key!

• Add a subkey: Subkeys enhance security. Use gpg --edit-key [Your Key ID] followed by addkey and save to add one.

• Display subkeys: After listing keys with gpg --list-keys, edit a key using gpg --edit-key <Your Key ID>. Copy the subkey ID and use gpg -a --export <KEY ID> to display it.

Sharing and Revoking Keys

• Send your public key to a public server: Public keys are meant to be shared. Use gpg --send-keys <Your KEY ID> to upload it.

• Generate a revocation certificate: If your key is compromised, create a revocation certificate with gpg --output revoke.asc --gen-revoke <Your Key ID>. Import it and upload it again to revoke the key.

Searching for and Importing Public Keys

• Search for a public key: Find a key using its ID with gpg --search-keys <KEY ID>.

• Import a public key: Once found, import the key with gpg --recv-keys <Key ID>.

Understanding Trust

• Beware of key impersonation: Anyone can create a key with your email address. To verify a key's authenticity, compare fingerprints.

• Verify fingerprints: Use gpg --fingerprint <Person's KEY ID> to see a key's fingerprint. Contact the person and confirm it matches.

The Web of Trust

The web of trust allows you to trust others based on established trust relationships.

• Trust a key: Use gpg --edit-key <Person's Key ID>, then trust and save to trust someone's key.

• Sign a key: Signing a key verifies its ownership. Edit the key with gpg --ask-cert-level --edit-key <Person's Key ID>. Look for "Full" trust on the left side of the user ID. Use check to see who signed the key and sign to add your signature. Upload the signed key again to the public server.

• Revoke your signature: If a signed key becomes invalid, use gpg --edit-key <Person's Key ID>, then revsig and save to revoke your signature. Upload the updated key information.

Encryption vs. Signing

• Encryption uses the receiver's public key to scramble the message. Only their private key can decrypt it. Share your public key beforehand for them to receive encrypted messages.

• Signing uses your private key to create a digital signature that verifies the message's origin and integrity. The receiver uses your public key to confirm the signature.

Using GPG with Git

• Signed commits: Add your GPG public key to your GitHub account.

  Go to: https://github.com/settings/keys

  Scroll to the bottom. You'll find Add GPG Key button in the GPG keys section.

  Export your public key using this command:

  gpg --export --armor <KEY ID>

  Replace <KEY ID> with your actual KEY ID which you can find by listing your keys using this command: gpg --list-keys.

  If you have not created a gpg key yet, check this out: Generating and Managing Keys

  Copy the exported public key content and paste it in the Github and save.

  Note: The email associated to your key should match the primary email in your github and email in your local git.

• Sign a single commit: Use git commit -S <Your Key ID> -m "message" to create signed commits.

• Global configuration: Set global Git settings with:

  • git config --global user.signingkey "<Your KEY ID>"

  • git config --global commit.gpgsign true

  • git config --global tag.gpgsign true

• Commit email: Ensure the primary email in your key matches your Git commit email. Set it with:

  • git config --global user.email "<Email associated to your Key>"

• Verify signed commits: Use git log --show-signature to see which commits are signed.

Using GPG to Encrypt Emails

GPG keys can encrypt emails. Upload your private key to a trusted email client like Thunderbird. You can then choose to digitally sign or encrypt emails based on the recipient's public key availability.

What are iDRAC and LCC?

Let's start with the basics: what exactly is this iDRAC? Well, picture this – you've got your shiny new Dell PowerEdge R6525 servers unboxed and powered up, but how do you manage them without being glued to the server room 24/7? That's where iDRAC comes in.

The Integrated Dell Remote Access Controller (iDRAC) is designed to enhance the productivity of server administrators and improve the overall availability of PowerEdge servers. iDRAC alerts administrators to server problems, enabling remote server management, and reducing the need for an administrator to physically visit the server. iDRAC with Lifecycle Controller allows administrators to deploy, update, monitor and manage Dell servers from any location without the use of agents in a one-to-one or one-to-many method. This out-of-band management allows configuration changes and firmware updates to be managed from Dell EMC, appropriate third-party consoles, and custom scripting directly to iDRAC with Lifecycle Controller using supported industry-standard API’s.

Now, let's talk setup. Getting iDRAC up and running is a crucial first step.

Getting Started: Setting Up iDRAC and LCC

We have three Dell PowerEdge servers at our disposal. I'll walk you through the setup process for just one server. But, no need to worry. The procedure remains identical for the other two. The only variation lies in the IP addresses, which will be unique to each server.

First off, we need to give our server the network details it needs for remote access – think IP and Gateway. During the initial setup phase, our physical presence and direct connection to the server are required. Once this setup is complete, we gain access to iDRAC through a web interface. Simply enter the provided IP address into your browser, and you'll be greeted with a user-friendly interface, resembling the image below.

Once that's sorted, it's time to ensure we're up to date. It is very important to keep the server firmware updated to ensure our server is secure and performant. Heading over to the Dell website's support page, we can find the drivers and downloads section for our specific model. Here, we're interested in the BIOS and iDRAC with Lifecycle Controller updates.

Once we've got the firmware files downloaded, it's time to check our current version. Logging into iDRAC, we navigate to the Virtual Console, then the Lifecycle Controller, where we can view our current firmware versions. If there's a newer version available, it's as simple as initiating the upgrade process.

I won't explain the upgrade process here, because it's fairly straightforward. You can upgrade system using iDRAC Web Interface.

Now, let's get back to the Lifecycle Controller (LCC). This handy feature is like a mini operating system embedded in our Dell servers, allowing us to manage them remotely via iDRAC. It's only booted up when needed, keeping resource usage to a minimum.

When setting up our servers initially, we'll also have the option to configure the network for the Lifecycle Controller. While it's not mandatory, doing so enables us to remotely upgrade drivers from any server within the network.

Just remember, the IP addresses for iDRAC and LCC cannot be the same, nor can they match the OS IP. However, the LCC and OS IPs can be the same, given they're not booted up simultaneously.

And there you have it – the ins and outs of setting up iDRAC and Lifecycle Controller for efficient server management. Stay tuned for more insights as we continue our HCI journey!

Hey there, tech enthusiasts!

In our ongoing journey of building a hyper-converged infrastructure (HCI), we've reached an exciting phase where we're diving into configuring high availability (HA) and link aggregation in our Ruijie Company Switches. These crucial steps will not only enhance the reliability and performance of our infrastructure but also ensure seamless operation even in the face of failures.

Now, before we dive into the nitty-gritty details, I want to mention that we'll skip over the basic setup tasks like assigning IP addresses and enabling SSH. Instead, let's jump straight into the action!

Configure Virtual Switch Unit (VSU)

In Ruijie switches, the concept of Virtual Switch Unit (VSU) is akin to switch stacking in other vendors' switches like CISCO. It allows us to virtualize multiple physical switches into one logical unit for simplified management and enhanced redundancy. Here's how we set it up:

First, we'll configure Virtual Switch Unit (VSU) on each switch. This helps combine multiple switches into one logical unit for easier management. Each VSU gets a unique domain ID, but multiple switches can share the same ID if they're part of the same VSU.

Next, we'll assign a priority number to each switch. This number determines which switch is the main one (active) and which is the backup (passive). The active switch handles configuration changes, while the passive one waits in case the active switch goes down.

Then, we'll configure the interfaces for the Virtual Switch Link (VSL). The number of interfaces needed depends on the switch model, usually between 2 to 8. You can check your switch's configuration guide for details. Here, I've used Ten Gig Interfaces 23 and 24 from both switches.

Finally, we'll switch the mode to virtual. This process backs up the old standalone switch configuration and reloads it when you remove the VSU setup later on. It's like putting the switch into a special mode where it behaves as part of a larger virtual unit.

On Switch 1:

sw_A> enable
sw_A# configure terminal
sw_A(config)# switch virtual domain 100
sw_A(config-vs-domain)# switch 1
sw_A(config-vs-domain)# switch 1 priority 200
sw_A(config-vs-domain)# switch 1 description sw_A
sw_A(config-vs-domain))# exit
sw_A(config)# vsl-port
sw_A(config-vsl-port)# port-member interface tengigabitethernet 0/23
sw_A(config-vsl-port)# port-member interface tengigabitethernet 0/24
sw_A(config)# exit
sw_A# switch convert mode virtual

On Switch 2:

sw_B> enable
sw_B# configure terminal
sw_B(config)# switch virtual domain 100
sw_B(config-vs-domain)# switch 2
sw_B(config-vs-domain)# switch 2 priority 150
sw_B(config-vs-domain)# switch 2 description sw_B
sw_B(config-vs-domain))# exit
sw_B(config)# vsl-port
sw_B(config-vsl-port)# port-member interface Tengigabitethernet 0/23
sw_B(config-vsl-port)# port-member interface Tengigabitethernet 0/24
sw_B(config-vsl-port)# exit
sw_B# switch convert mode virtual

Configuration VLAN

If you're here after reading my previous article, you might recall that we're planning to use separate VLANs for VM migration and CEPH. Now, let's dive into configuring those. If you haven't had a chance to read the previous article yet, you can find it here.

virtual_switch(config)# vlan 4
virtual_switch(config-vlan)# name migration
virtual_switch(config-vlan)# exit

virtual_switch(config)# vlan 5
virtual_switch(config-vlan)# name storage
virtual_switch(config-vlan)# exit

Configure Link Aggregation (LAG)

For simplicity, I'll demonstrate the LAG configuration for just one aggregate port, which we'll call aggregatePort3. However, in our setup, we've actually configured LAG for other aggregate ports as well, specifically aggregatePort 4, 5, 9, 10, and 11.

Link Aggregation is essential for increasing throughput and ensuring redundancy by combining multiple physical links into a single logical link. Let's walk through the steps:

1. Create a Port Group: Assign two ports from different switches to a group.

// Select an interface of switch 1
virtual_switch(config)# interface tenGigabitEthernet 1/0/3
virtual_switch(config-if-TenGigabitEthernet 1/0/3)# port-group 3 mode active 
virtual_switch(config-if-TenGigabitEthernet 1/0/3)# exit

// Select an interface of switch 2
virtual_switch(config)# interface tenGigabitEthernet 1/0/3
virtual_switch(config-if-TenGigabitEthernet 1/0/3)# port-group 3 mode active  // 3 is group ID 
virtual_switch(config-if-TenGigabitEthernet 1/0/3)# exit

Here, we've set up an active-active configuration. This means we're maximizing both throughput and failover safety.

2. Show Aggregate Port Status: Check the status of the aggregate port.

virtual_switch(config)# show aggregatePort 3 summary
// You can see switchport mode is access

3. Convert to Trunk Mode: Convert the access mode to trunk. Ruijie switches, LAG operates exclusively in Trunk mode and requires explicit setup.

virtual_switch(config)# interface aggregatePort3 
virtual_switch(config-if-AggregatePort 3)# switchport mode trunk
virtual_switch(config-if-AggregatePort 3)# switchport trunk allowed vlan add 4,5
virtual_switch(config-if-AggregatePort 3)# end

4. Save Configuration: Ensure all configurations are saved for persistence across reboots.

virtual_switch# write

And there you have it! With VSU for high availability and LAG for improved link utilization and redundancy, our hyperconverged infrastructure is becoming more resilient and efficient by the minute.

Stay tuned for more updates as we continue to fine-tune our setup and explore new horizons in the world of IT infrastructure.

Keep exploring!

As an IT Infrastructure Engineering Intern, I've had the incredible opportunity to dive headfirst into the world of hyperconverged infrastructure (HCI). It's been a whirlwind of learning, problem-solving, and hands-on experience, and I couldn't be more excited to share my journey with you.

Let's start by breaking down the components of our setup. We're working with two Ruijie Company Switches, specifically the RG-CS86-24XMG4XS4VS-UPD model. These bad boys pack quite the punch with an ARM processor clocked at 1.25 GHz, 1GB of flash memory, and 1GB of SDRAM. With a plethora of ports ranging from 100M to 10GE, including support for PoE/PoE+/PoE++ and modular power supply and fan slots, these switches are the backbone of our infrastructure.

But what good are switches without servers, right? That's where the Dell Poweredge R6525 servers come into play. We've got three of these powerhouses ready to rock and roll. Equipped with powerful processors, ample memory, and storage capabilities, these servers are the muscle behind our HCI setup.

Now, let's talk about the heart and soul of our hyperconverged infrastructure: the software. We've chosen Proxmox VE as our hypervisor and CEPH for cluster storage. Proxmox VE provides us with a robust virtualization platform that's not only easy to manage but also offers high availability and scalability. And with CEPH handling our storage needs, we can ensure data redundancy, fault tolerance, and seamless scalability across our cluster.

Here is a simple Network Diagram, what we are going to cover in the next few articles.

Let me explain the network diagram to you. We'll mostly skip the Fortinet part for now, as it's not relevant to this series.

Here's the bigger picture of what we'll do:

We'll stack two switches and configure Link Aggregation. We'll define aggregatePort 3, 4, and 5 for server internal communication and aggregatePort 9, 10, and 11 for server management. The links from aggregate port 3, 4, and 5 will connect to the ten-gigabit interfaces of the servers, while links from aggregate port 9, 10, and 11 will connect to the one-gigabit interfaces of the servers. We'll use interface 1 for the uplink and the management port as the management interface.

To manage the Dell server remotely, we'll set up iDRAC. For iDRAC, we'll connect the Fortinet and iDRAC port directly.

On the server, we'll install Proxmox VE, which is a KVM-based open-source hypervisor. We'll then create a cluster of all three nodes. After that, we'll set up Ceph for storage clustering.

Also, we'll create two bonded interfaces: one for server internal communication, which will consist of two ten-gigabit interfaces, and another for management, which will consist of two one-gigabit interfaces.

Additionally, we'll set up VLAN 5 for Ceph and VLAN 4 for VM migration.

Are you excited ??

As we embark on this journey, I'll be sharing my insights, challenges, and triumphs with you all. From configuring our switches and servers to setting up our hypervisor and storage cluster, there's a lot to cover. But fear not, I'll do my best to explain the technical jargon along the way without overwhelming you.

So buckle up, folks! We're about to dive deep into the world of hyper-converged infrastructure, and I can't wait to take you along for the ride.

Copy on Write (COW) vs. In-Place File Systems

At the core of the distinction lies the mechanism of handling write operations:

• Copy on Write (COW): In a COW file system, modifications are executed by creating entirely new blocks. These modifications are then linked to the metadata pointers, ensuring atomic writes. This approach is write-intensive but mitigates the risk of corrupted blocks while taking snapshots. We can take snapshots at any point in time without freezing I/O.

• In Place File Systems: Conversely, in an In Place File System, modifications occur by updating existing blocks. These changes are recorded in a journal, facilitating recovery in the event of data corruption due to unforeseen disasters. While less write-intensive, in-place filesystems are not consistent at every point in time. So, to take snapshots of in-place file systems we have to freeze I/O to ensure the snapshot data is not corrupted.

Examples: Notable COW file systems include ZFS and BTRFS, while XFS and EXT4 represent In Place File Systems.

ZFS: Renowned for its advanced features, ZFS encrypts data on write and offers capabilities like storage pooling, scalability, encryption, snapshot backups, and compression. However, these features demand higher CPU and RAM resources.

BTRFS: Although not as mature as ZFS, BTRFS shares similar characteristics and drawbacks. Its snapshot functionality isn't as robust as ZFS's.

EXT4: As one of the most mature and fastest file systems, EXT4 is widely used but lacks the advanced features of its counterparts.

XFS: Preferred by industry leaders like RedHat, XFS is fast, scalable up to exabytes, and suitable for enterprise environments.

Selecting the optimal file system depends on specific use cases and requirements. If you prioritize performance over. In our use case, we opted for the XFS file system. The decision to opt for XFS over ZFS or BTRFS was driven by the requirements of setting up an HCI (Hyperconverged Infrastructure) where CEPH serves as the primary storage option. With CEPH providing all the advanced features of ZFS and BTRFS, the focus shifted towards a faster and more stable file system, making XFS the ideal choice for the task at hand.

In Conclusion

Understanding the nuances between Copy on Write and in-place File Systems empowers us to make informed decisions regarding data management strategies. Whether prioritizing performance, scalability, or disaster recovery, the choice of file system plays a pivotal role in shaping the efficiency and resilience of our storage infrastructure.

Since the COVID-19 pandemic, more and more people are working remotely. It has become a popular choice for organizations and individuals. But there's one big problem: trust. Companies can't always be sure if their remote employees are working or just slacking off. To solve this, many use time tracker software. However, the problem is that employees don't trust these tools. They worry about their privacy and how their data is being used. That's why I came up with an idea for a project: creating an open-source time tracker.

As a freelancer who works remotely, I faced the same trust issues. Existing time trackers just didn't cut it. They were closed-source and made me worried about how my data was being handled. Most of them ran silently in the background, taking screenshots and tracking my keystrokes. This raised privacy concerns and made me even more skeptical. It was clear that we needed a more transparent approach to time tracking.

That's where my project "Scout" comes in. I developed an open-source time tracker using Appwrite Cloud for Appwrite Cloud and Hashnode Hackathon, that fosters transparency and trust. By making it open-source, both companies and employees can have a say in how it's built and ensure that it's trustworthy. With open-source software, anyone can look at the code to make sure there are no hidden features compromising privacy or misusing data.

Author

• Aadarsha Dhakal - @aadarshadhakalg

Description of Project

Scout is a desktop application that currently supports the GNU/Linux operating system. To access Scout, users can easily authenticate using their phone number, and the process is quick and hassle-free.

With Scout, users can send or receive contracts. A contract represents the details of a specific job or position, whether it's a project in freelancing or a corporate job role. Before starting work on a contract, the receiver must accept it. If the receiver is not interested in working on the project, they can simply decline the contract.

Likewise, the sender of the contract has the ability to either end or hold the contract if needed.

When it comes to tracking work hours, employees can clock in to start recording their time using the tracker feature. To ensure that the client is actively working on the project, the tracker will randomly capture screenshots. Once the employee has finished working, they can clock out. They can have multiple such work sessions, and all the records of these sessions are securely saved in the Appwrite cloud database. Both the contract employer and the employee can see the work diary of the contract. The work diary contains the list of the work sessions and the screenshots captured. The screenshots taken during work sessions are stored in the Appwrite cloud storage for easy access and reference.

Tech Stack

• Flutter

  • flutter_bloc

  • get_it

• Appwrite Cloud

  • Authentication

  • Database

  • Storage

Scout uses Flutter for the frontend and Appwrite Cloud for the backend. It utilizes Appwrite's phone authentication for secure login and its database and storage features to store user data and screenshots

Appwrite's amazing support for Flutter through the package made it an excellent choice for Scout.

Challenges I Faced

During the development of this app, I encountered a few challenges. The primary hurdle was the lack of an existing package in Flutter that could capture desktop screenshots in Linux. To overcome this, I had to develop my own plugin. You can find a blog post detailing the process and development of the plugin.

Plugin: ScreenshotX

Blog: Interacting with System Services using DBus Dart

I spent a significant portion of my time and effort in building this plugin. As a result, the remaining time for project completion was limited, and I started to worry about meeting the deadline.

However, this is where Appwrite proved invaluable. With its comprehensive backend features, I was able to leverage its capabilities and complete the project on time, despite the limited timeframe.

Public Code Repo

https://github.com/aadarshadhakalg/scout

Demo Link

https://youtu.be/bOMTwdCgdt4

Installation Sucks

Disclaimer: RedHat is one of the greatest contributors to the Linux Kernel. And RedHat Linux is mainly created for servers and high-performance computing. So, using it on a personal computer was never going to be a good idea. But, what's wrong with trying right?

I inserted the USB, hit the power button, and began the installation process. The installation flow was very confusing. I had to go back and forth between some screens and it was not so smooth. You don't see what needs to be done next. And, it's very likely that you miss some steps.

Things you need to do to install RedHat and other Linux are not the same. You don't have to log in to your account and activate your subscription in the other Linux. You don't see software selection and the Kdump thing in other Linux installations. I had to perform installation twice because I chose 'Workstation' at first and it didn't have a GUI. Later, I choose 'Server with GUI'. And the hardest part was choosing a security policy. There were so many security policies that it was a very hard choice to make. But hey, you believe I am a genius right? Yes? You can say that again!

I chose NONE. And proceed with the normal installation. Since it was a network install, I had to make sure that the internet connection was fast.

Red Like Rose

The desktop look is beautiful. RedHat comes with Gnome Desktop Environment by default. The wallpaper was looking so nice with the RedHat Enterprise Linux branding on the bottom left of the desktop. RedHat used Wayland for display management. It also supports the touchpad gestures to reveal menus and switch desktops. You can do a certain level of customizations using gnome-extensions and gnome-tweak-tools. It's nothing out of the world but it gives the 'Linux Vibes'.

RedHat Linux comes with the nouveau driver for Nvidia Graphics Card. But, it can be easily replaced with the official Nvidia proprietary driver.

Overall, The UI feels so smooth, Looks nice in red, so the GUI experience was good. But it didn't last long.

The Game of 'Thron'

It's not all sunshine and rainbows. I was enjoying the looks but the real pain was waiting for me. You can feel the real pain of RedHat Linux as a daily driver once you step into package installation. I couldn't find the most essential software in the RedHat package repository. For this, I had to add and enable multiple third-party repositories but still, lots of packages were missing. There is no promise that you'll get the packages you need. I had to install snap to download very common applications like Spotify, Intellij, VS Code, GIMP, and Inkscape. This shows how painful it is to install the software in RedHat Linux. There is no straightforward way to install applications and for every app to install it's very possible that you'll either fail completely or the way to get it is jagged.

Also, I don't know why but my Bluetooth was not functioning properly. Every time I want to connect a new device or reboot my system, I had to restart the Bluetooth service. Since it was working great with Arch Linux, I can say that device driver availability is also not good.

The Conclusion

All of these issues compelled me to replace the RedHat Linux with Pop OS. I am not saying Pop OS is the best distribution for your personal computer but RedHat is definitely not.

RedHat might be the best option for the servers; I think it is; as it offers high security, stability and very good tech support. From my 10 days of RedHat experience on my personal laptop, I can say that it doesn't worth a try.