Configuring a Linux environment for the team: a simple and efficient project sharing solution.

Preface

It's been a while since I last wrote a blog post—things have been incredibly busy lately. I've also started a fun new project; once it's further along, I'll write an article to introduce it.

In our current AI project, the team needs to share a GPU server for model training and data processing. To ensure every team member can use the server efficiently, we decided to set up a shared multi-user environment. This way, whether it's code development, model testing, or result verification, everyone can conveniently access and leverage the server's powerful computing resources.

This article will document the process of configuring the Linux shared environment, and hopefully, it will also help other teams facing similar needs.

Setting Up Users and Groups

To efficiently manage our GPU server, we first need to create new user accounts and organize these accounts into a dedicated user group. This approach simplifies permission management and access control.

Creating Users and Groups

First, create a group

sudo groupadd gpugroup

Next, create a user

sudo adduser [username]

Then add the user to the group

sudo usermod -a -G gpugroup username

Configuring SSH Public Key Login

To enable team members to securely connect to the server via SSH, public key authentication should be used uniformly, with password login prohibited.

Each user needs to generate a key pair (if they haven't already) and send the public key to the administrator. Then, the administrator needs to add these public keys to the user's .ssh/authorized_keys file.

First, ensure that each user's home directory has a .ssh directory. If it doesn't exist, you can create it using the following command:

sudo mkdir /home/[username]/.ssh
sudo chmod 700 /home/[username]/.ssh

Next, add the public key to the authorized_keys file:

echo [public key content] >> /home/[username]/.ssh/authorized_keys
sudo chmod 600 /home/[username]/.ssh/authorized_keys

Replace [username] and [public key content] with the actual username and public key.

Creating a Shared Folder

Previously, I stored my code in the home directory. However, in practice, I found this to be unfavorable for sharing.

After all, designating a folder under a user's home directory as a shared directory for other users to access always feels a bit odd.

Typically, for better security and organization, it is preferable to create a dedicated directory for shared projects. I consulted GPT, and here's what it advised:

Security: Avoid unnecessary permission exposure. When you place projects within a personal home directory, you might inadvertently grant other users excessive access rights, which could lead to sensitive information leaks or accidental data corruption.

Manageability: A dedicated shared directory makes management more straightforward. You can easily control who has access to this directory without worrying about affecting other personal data or settings.

Scalability: As the team grows, more users may need access to these shared resources. Having an independent shared directory simplifies managing users and permissions, instead of constantly adjusting permission settings under the home directory.

Clear Permission Demarcation: An independent shared directory makes setting and adjusting permissions clearer and simpler. For instance, you can apply customized security policies and backup strategies specifically for this directory.

Therefore, I created a directory /srv/projects under the /srv directory to use for sharing.

Group Permission Settings

Next, set the ownership of the team-shared project folder /srv/projects to the gpugroup group.

sudo chown -R root:gpugroup /srv/projects
sudo chmod -R 775 /srv/projects

Setting the SGID (Set Group ID)

SGID is a special permission setting that ensures any new file or directory created within a directory automatically inherits that directory's group. This is ideal for our project directory because it allows all team members to access and modify files without worrying about individual file group settings.

You can use the following command to set the SGID bit:

sudo chmod g+s /path/to/directory

For example, if our project directory is /srv/projects, the command would be:

sudo chmod g+s /srv/projects

With this setting, all new files and directories created under /srv/projects will automatically be assigned to the group, maintaining consistent permissions.

Adjust umask (optional)

PS: I didn't set this, it's possible but not necessary

umask is a system setting that determines the default permissions for newly created files and directories. To ensure team members can edit each other's files, we need to set an appropriate umask value.

The usual umask value is 022, which means new files default to permissions 644 (user can read and write, group and others can read), and new directories default to permissions 755 (user can read, write, and execute; group and others can read and execute). For team collaboration, we can set the umask to 002, so new files have permissions 664 (user and group can read and write, others can read) and new directories have permissions 775 (user and group can read, write, and execute; others can read and execute).

You can temporarily change the umask value with the following command:

umask 002

For permanent changes, you need to set it in the user's shell configuration file, such as .bashrc or .profile:

echo "umask 002" >> ~/.bashrc

Other Notes

There are some additional details. For example, I previously installed conda in the home directory.

Now I need to reinstall it in /srv/apps, and the original installation in home is no longer needed.

This way, all users can share Python virtual environments.

Displaying Progress When Copying Files

The cp -R command does not support displaying progress.

You can use rsync or pv as alternatives.

rsync

rsync -ah --progress source destination

Parameter Explanation:

• -a is archive mode, which preserves symbolic links, file permissions, user group information, etc.
• -h makes the output easier to read by using a human-readable format.
• --progress displays the copy progress.

pv

This is used in conjunction with compression tools and may not be pre-installed on some systems, requiring installation first.

tar cf - source/ | pv | tar xf - -C destination

This command archives the source directory and uses pv to display progress, then unpacks it in the destination directory.

Summary

This setup is basically done.

I've also recently explored some project management tools and deployed MatterMost.

Next, I can write articles to document this.