Is it possible to set CloudWatch alarm for a combination of metrics?

For the last overall goal, I would set the alarm on ApproximateNumberOfMessagesVisible from Cloudwatch documentation on SQS this is:

  The number of messages available for retrieval from the queue.

  Units: Count

  Valid Statistics: Average, Minimum, Maximum, Sum, Data Samples
  (displays as Sample Count in the Amazon SQS console)

For the more generic computed metric question I would go with a lambda running priodically, gathering metrics from cloudwatch, doing the maths and pushing back to cloudwatch.

Here is an example lambda gathering data from Cloudwatch to push to elasticsearch. With the examples on the documentation you should be able to push metrics back from the lambda.

Be aware that calling cloudwatch API to gather and push metrics has a cost when you got over 1M request per month, there's other limits to be aware of to avoid breaking your budget/workflow.


Triggering specific pipeline builds for monorepos in Jenkins

You can use the "when" block combined with the built in "changeset" condition to conditionally run only certain stages of your monorepo's pipeline. From the when.changeset documentation:

changeset-
Executes the stage if the build’s SCM changeset contains one or more files matching the given string or glob. Example: when { changeset "**/*.js" }

Here is an example Jenkinsfile using this strategy:

pipeline {
    agent any
    stages {
        stage('build matchengine') {
            when {
                changeset "**/matchengine/*.*"
            }
            steps {
                echo 'building match engine'
            }
        }
        stage('build posttrade') {
            when {
                changeset "**/posttrade/*.*"
            }
            steps {
                echo 'building post trade'
            }
        }
    }
}

, applicable to the monorepo project structure shown below:

 .(my-project)
   |-- Jenkinsfile
   |-- matchengine
   |-- posttrade
   |-- serverless
   |-- ui

This strategy won't scale past small codebases because it would be hard to keep track of which modules depend one each other. You would be better off using a build system like Bazel. Your CI job would simply issue a bazel build //... (build everything), and Bazel would calculate what actually needs to be built, and what needs to be tested. Further, there even exist bazel rules such as rules_docker and rules_k8s which can calculate which of your containers need to be rebuilt and pushed to a container registry, and which of your applications need to be redeployed to Kubernetes.


Ansible how to know what return value is provided by a specific module

if you do something like:

- name: Launch instance
  ec2:
  register: data_struct

You can then output all the values returned by doing this below the above code:

- name: print all returned values
  debug:
    msg: "{{ data_struct }}"


Preventing accidental production deployments

The best practice is to separate production and staging(testing) inventory.  See Staging vs Production.

  This way you pick with -i what you are targeting.

To make it more robust you might want to clone a separate structure for production and restrict access to it. Pull from staging before deploying to production.

In detail, this would mean to 1) put the directories into git, 2) make sure staging is committed, pushed and merged, and 3) then pull to the production and deploy. The workflow is up to you.


One argument in support of storing artifacts is that it promotes consistency when deploying across different environments.
From Continuous Delivery by Humble and Farley:

Every time you compile the code, you run the risk of introducing some difference. The version of the compiler installed in the later stages may be different from the version that you used for your commit tests. You may pick up a different version of some third-party library that you didn’t intend. Even the configuration of the compiler may change the behavior of the application.

By only building artifacts once, we make sure that the production code is exactly the same as that which has gone through testing. In very small codebases with simple pipelines, it might not make too much of an impact, but it's still best practice to only build once.


No.  The account that uploads an object into a bucket owns the object.  All the bucket owner account can do, unless granted permission by the object owner account, is delete the object.


What are best practices to override complete sections using docker-compose?

According to this documentation it does not seem to be possible to override complete sections in docker-compose by using different docker-compose files.

  If both files are present on the same directory level, Compose
  combines the two files into a single configuration.

  The configuration in the docker-compose.override.yml file is applied
  over and in addition to the values in the docker-compose.yml file.


No, you don't need maven.
Maven is a build automation tool as ant and gradle are, generally, but not limited to, jvm based projects.
Nexus is an artifact repository, but some others exists.
Artifact repositories software have different set of features (license, different storage options, different repositories type support like gems or npm ..., search support, web interface ...).

I know that once a developer pushes a code to git, Jenkins should take it and run the test case on it.
If it is success then push it to production? Am I right? I'm not sure about the flow.

Generally, in order to achieve continuous deployment, which is what you are asking for precisely here, you would:

setup a hook when you commit
have a continuous integration tool that receive the hook
the CI does whatever needs to be done for the project to be tested / build / runnable
the CI deploys the previously built software

Jenkins, for example, supports pipeline as code, it would allow you to put a Jenkinsfile in your repository, that would tell Jenkins how to do step 3 and 4 (travis and circleCI and others CI also supports this).
You can tell your CI in steps 3 to upload some artifacts to an artifact repository and on step 4, to retrieve them in order to deploy the software, but you don't need to.
My advice would be to design your CD system the simplest way you can, don't introduce not necessary parts like an artifact repository if you don't need it, add it when you face a problem that needs it.
As a first step, your CI, can during the deployment steps, simply copy your artifacts to the target server.


Build / Deployment process for .Net/Mulesoft/Oracle stack

For something that requires being agnostic to the tooling, and having the flexibility you need, your best bet is Jenkins.  https://jenkins.io/.  They have a ton of plugins that would meet any use case.  Be aware that Jenkins can be heavy for small teams, but your other options lack the flexibility and support for different communities.

You can check out other tools:

Team city
CircleCI/Travis CI
Semaphore
ETC


All container services and Function-As-a-Service services of the big cloud providers, such as Amazon ECS, Lambda and Azure cloud functions run on actual virtual machines of these providers. Normally they only take a small number of milliseconds to start up, but when your container or function has not run for a certain amount of time, they will take a couple of seconds to actually start up the virtualization environment for your container or function. These startup times improve all the time, but you still get better startup time results by calling your functions or a request in your container in regular intervals.

A typical interval could be every minute, so that you can be sure that the virtual environment the container or function runs on has likely not been shut down, as it just was in use in the past 60 seconds.

In relation to an auto scaling group, I would say that "warm up" is a bit of a mis-nomer. Either there are enough instances provisioned to handle the traffic you expect, or you may need to make a 'scheduled change' to the number of instances, but there is actually nothing to warm the instances up.

One edge-case would be if instances have disk contents restored from backup. On AWS, there is some delay in getting the contents on these disk from snapshot and the disk can be warmed up by reading all blocks on it


It supports AccuRev, CVS, Subversion, Git, Mercurial, Perforce, Clearcase and RTC.

For all of them exists a plugin and as you probably already know, Jenkins is not limited to only that list, anyone can create a SCM plugin for other options if they want to.

Here is the link to the plugins:

AccuRev: https://wiki.jenkins-ci.org/display/JENKINS/AccuRev+Plugin
CVS: https://wiki.jenkins-ci.org/display/JENKINS/CVS+Plugin
Subversion: https://wiki.jenkins-ci.org/display/JENKINS/Subversion+Plugin
Git: https://wiki.jenkins-ci.org/display/JENKINS/Git+Plugin
Mercurial: https://wiki.jenkins-ci.org/display/JENKINS/Mercurial+Plugin
Perforce: https://wiki.jenkins-ci.org/display/JENKINS/Perforce+Plugin
Clearcase: https://wiki.jenkins-ci.org/display/JENKINS/ClearCase+Plugin
RTC: https://wiki.jenkins-ci.org/display/JENKINS/Team+Concert+Plugin


Why are valid directives in my Dockerfile apparently being ignored?

It looks like your Dockerfile is attempting a multistage build:

FROM ubuntu:latest
...
FROM python:2.7
...

This is a feature that enables you to build parts of your image in a separate build container and then import what you need. Unless you import stuff into your end container, it will be lost in the build process; this is exactly what is happening with your RUN apt-get... statement. To fix it, try removing the extra stages of the build, (FROM ubuntu:latest and FROM python:2.7) and moving your commands to after FROM cassandra:3.11.4 like so:

FROM cassandra:3.11.4

USER root

RUN apt-get -y update \
    && apt-get -y upgrade \
    && apt-get install -y \
        python-pip \
        nmap \
        vim-tiny

RUN pip install --no-cache-dir cassandra-driver
...

I'm sorry you're having a hard time with Docker. Hopefully this helps and it starts making more sense. Good luck, friend!


How to get CPU and Memory usage of nodes in a specific Kubernetes instance group?

Do your instance groups have labels identifying the groups they belong to? If so, you can filter to them using the -l/--selector option, like so:
kubectl top nodes --selector instance-group=group-1

This assumes that the label name is instance-group and the label value for the group you want to query is group-1.


Trigger jenkins pipeline job with Bitbucket hook

If you just need generic post-receive hooks, you can just do a regular web hook and hit the build API endpoint.  For instance, if the name of your Jenkins server is jenkins.example.com and the name of the Pipeline job to trigger is my_awesome_job, then you can fire off a POST request to https://jenkins.example.com/job/my_awesome_job/build.  Depending on your security model, this request may or may not need to be authenticated and authorized.  If you do need authentication and authorization, you can use HTTP Basic auth (unfortunately I haven't found a reasonable way to do keypair auth).

If you need to trigger builds on pull requests, then you'll need a plugin. I've been using the pull-request-notifier plugin for Bitbucket to trigger builds from Bitbucket and it's worked pretty well.  It works especially well in combination with the Bitbucket Branch Source plugin for Jenkins, which automatically creates Jenkins jobs for every branch and open PR of every single repository in a Bitbucket project.  The combination of these two plugins allows me to automatically trigger builds as soon as a PR is opened and prevent merging the PR until the build passes (this is a pretty common workflow so I figure it's worth mentioning even if you didn't explicitly ask about it).


Service backend/frontend in one or two containers?

Are they separate projects (from a logical or technical perspective)? Would one ever need to be updated/recreated independently of the other one? Would they need to scale independently of each other? I think if any of those answers is a yes, then they should be separated. If not, you should be fine bundling them together. Bundling them together will reduce discrepancy or compatibility issues between the two containers.


weave-net is trying unsuccessfully to talk to the apiserver at 10.32.0.1 (port 443). Verify your master node is indeed at that IP address, it has network connectivity to the node where this is barfing, and that the firewall on the master node has port 443 open.

If they are all running on the same host, verify the apiserver is running.


How do I configure Travis CI build testing for a Java Eclipse project?

The easiest method is to have Eclipse generate an Ant build script for you.

Right click on your project in the Package Explorer and select Export in the context menu. Choose the export type General -> Ant Buildfiles and click Next.

On the next screen, make sure your project is selected. You can keep leave the options on their default settings. Click Finish to generate your build.xml file.

Lastly, to ensure Travis CI runs the project correctly, create a .travis.yml in your project's root folder. For Java projects, it should contain at least the following.

language: java
jdk:
  - oraclejdk8

script: ant build

Source: Kofun devblog - Using Travis CI with Java Eclipse Project


Help! No development infrastructure in company! I've been tasked to implement it!

Have you been in this scenario before?

Yes. Pretty standard scenario in my experience.

  How did you address and overcome these obstacles?

In very small steps.

  How long did it take to implement? (as a rough idea)

Very long; in one rather large extreme case: over 10 years, and still ongoing.

  How would you prioritize the tasks listed above?

My priority, with additional comments:

Throw all active projects into source control.

This should really be a no-brainer and low-hanging fruit.

Implement coding standards.

Sensibly.

Introduce a methodology more suited to how the team currently operates, most likely Agile or Scrum and the tools and documentation templates that go with it.

If there is no methodology at all right now, then I'd start with a very lenient Kanban first - which basically just means throw up a ticket system and keep track of everything relentlessly. I would hold off with full-fledged Scrum until everything is on track and you're established as a Good Guy.

Implement staging so that we have Dev, QA, UAT and Live.

With this goes more separation; i.e. devs need to be weaned off of the production systems.

Introduce tooling and setup training for development team.

Try to have the teams develop their own tools if at all possible.

Implement CI/CD
Bring all projects up to using the same technologies.

Might just not be feasible, if by technologies you mean languages and libraries. If you mean Docker or OpenShift or whatever, then sure.

The most important tip for you from my experience: avoid, at all cost, to do everything yourself. Give "impulses" to the teams, but they have to do it (whatever "it" is) themselves. If you do all the work (either personally, or with your separate "DevOps team"), then you will, at the end of the day, not gain that much as everybody will just depend on you constantly.


How many iops do I actually get in a General Purpose SQL Database in Azure?

Follow this article
But in serverless mode you will have to look at limits for the vCores your DB is using at that second, so if vCores are 4 then you will have limit of 1280 and Throughput of '5-7 ms (write) and 5-10 ms (read)'


Vagrantfile change with Ansible playbook to control file permission on Vagrant synced folder for Trellis

There are two parts of this Vagrantfile, one does the mounting for Windows hosts (the first half), the other one for UNIX hosts (the second half), but essentially the do the same - just with different plugins.

You can see from the config.vm.synced_folder and config.bindfs.bind_folder commands where they set up the appropriate directories with the proper permissions.

To add a new directory with different permissions simply add them to the list. Note that File.join('a','b') simply converts the path to a/b, but it's a nicer way to do it, as it's platform independent (on Windows it can also handle \ style paths).

So for example if you want to give the /tmp/needswrite folder more permission you can simply add the following two lines to their appropriate places:

Windows config:

config.vm.synced_folder File.join(ANSIBLE_PATH, 'tmp', 'needswrite'), File.join(ANSIBLE_PATH_ON_VM, 'tmp', 'needswrite'), mount_options: ['dmode=777', 'fmode=777']

Unix config:

config.bindfs.bind_folder File.join(ANSIBLE_PATH_ON_VM, 'tmp', 'needswrite'), File.join(ANSIBLE_PATH_ON_VM, 'tmp', 'needswrite'), perms: '0777'

This will for example set their permission to 777 instead of the default 755

Note that in the Windows config you have to map from ANSIBLE_PATH to ANSIBLE_PATH_ON_VM, while on the Unix config bindfs will do a remount, so you need to match them with the same directory name.

You can also play with users and groups:

Windows config:

config.vm.synced_folder File.join(ANSIBLE_PATH, 'tmp', 'needswrite'), File.join(ANSIBLE_PATH_ON_VM, 'tmp', 'needswrite'), owner: 'new-owner', group: 'new-group', mount_options: ['dmode=755', 'fmode=755']

Unix config:

config.bindfs.bind_folder File.join(ANSIBLE_PATH_ON_VM, 'tmp', 'needswrite'), File.join(ANSIBLE_PATH_ON_VM, 'tmp', 'needswrite'), u: 'new-owner', g: 'new-group', perms: '0777'

Here we set them to use user new-owner and group new-group


How can I experiment with Cloud (Azure, AWS, Google, etc) without going broke?

The three major cloud platforms you mentioned have either free trials or a free tier; these all include spending and resource caps which prevent you from spending more than your free trial allows:

Azure

Azure offer £150/$200/€170 for the first one month of usage, in addition many of Azure's services have a free or low cost tier:

App Service Free Tier
SQL Azure Basic Instance @ £0.005/hour
Azure Machine Learning Free Tier
Azure Service Bus @ £0.04/million operations
Azure Functions @ £0.15 per million executions

If you have a MSDN or Visual Stuido Online account you can also get £100/$150/€130 per month of credit just because you are a subscriber to one of these services.

If you are a startup you may qualify for $120,000 of Azure Credits to host your application through Microsoft's BizSpark programme.

Amazon Web Services

Amazon have a Free Tier which includes:

750 Hours of EC2 which equates to about a month of a t2.micro instance
750 Amazon RDS
5GB S3
250,000 AWS IoT Messages
... and more

In addition there perpetually free usage allowances on most services:

AWS Lambda: 1 million free requests a month
DynamoDB: 25GB Storage
Amazon SNS: 1 million publishes
... and more

Google Compute Platform

Google offer a $300 free trial for 12 months in a similar deal to Amazon. In addition they have free usage tiers for many of their core offerings:

Google App Engine: 28 instance hours / day
Google Cloud Database: 1GB Storage
Google Compute Engine: 1 f1-micro instance / month
Google Pub/Sub: 10GB messages / month
Google Cloud Functions: 10 million executions / month
... and more

If all of that wasn't enough for you most of the cloud providers will compensate you in credits if you blog about their services in detail, you do have to put in a substantial effort up front and "appear" on their radar by writing blog posts, presenting at meetup and conferences but once they see you they will gladly put $100 a month towards your further learning.


Looking for a good book explaining how things work in the industrial system test environment and production environment

Dude, will depends of what you want to use or follow. Actually cloud + DevOps it's a buzzword in use.
So you can start reading about Cloud Architecture, Modern Infrastructure.
Refer about one book, maybe these: https://www.springer.com/gp/book/9783030037116


I would say It depend on you preference since redeploying a stack for only 1 or so image, will not restart all the containers present.

In the other hand network feature since docker 17.06 are reusable between stack with the attachable mode so you could easily separate the layers of your app while keeping them on the same sub-network (for usage like traefik or private data network).

Frankly for me this is the same problem as in dev when you separate the functions you are defining so it's easier for the others devs to understand a file of 50 lines compare to a file of 500 lines.

In the end it also depend on how this is integrated with your CD and if it's painful to update one document or multiple document when adding/removing services.

Personally I would go with multiples stack files.


This can be done with docker_registry_image:

data "docker_registry_image" "ubuntu" {
  name = "ubuntu:latest"
}

resource "docker_image" "ubuntu" {
  name          = "${data.docker_registry_image.ubuntu.name}"
  pull_triggers = ["${data.docker_registry_image.ubuntu.sha256_digest}"]
}


Parameterized Kubernetes manifest files?

I educated myself more about helm and it looks like it has these features that I need

packaging
templating
release management, including rollbacks

so I'll study it more.


What do you need for a monorepo for a large codebase?

The answer is: a bit of both. To satisfy the constraints of "use git" and "manage a vast codebase" Microsoft developed a new filesystem (previously they were using a variant of Perforce called SourceDepot). It's open source but I have no personal experience of using it.

Why would you want a monorepo? The most obvious reason is that you can modify an API and all the callers of that API in an atomic commit. Also there are advantages to being able to do a git log search across the entire codebase...


How do I configure SSH keys in a Vagrant multi-machine setup?

There is no general method and it might depend on how boxcutter/ol67 was packed.

The easiest method would be to define the password in the Ansible inventory file:

[oracle-vm:vars]
ansible_ssh_user=vagrant
ansible_ssh_pass=vagrant

The second method would be to leave the insecure private key configured on the oracle-vm machine and inject the private key to the ansible VM:

config.vm.provision "shell" do |s|
  ssh_insecure_key = File.readlines("#{Dir.home}/.vagrant.d/insecure_private_key").first.strip
  s.inline = <<-SHELL
    echo #{ssh_insecure_key} >> /home/vagrant/.ssh/id_rsa
    chown vagrant /home/vagrant/.ssh/id_rsa
    chmod 400 /home/vagrant/.ssh/id_rsa
  SHELL
end

Generate the key pair beforehand on the host machine, inject private key to Ansible VM, public key to Oracle's authorized_keys.
Generate the key pair on Ansible VM, copy the public key to Oracle VM using shell provisioner and inject vagrant as password for ssh-copy-id.

And the list does not end here, it depends on required security.


What Golang function could be used to enforce a minimum code coverage and return an Exit1 if false?

There aren't any off the shelf solution currently that allows you to specify a variable number of code coverage percentage.

There is the tool Courtney which enforces 100% code coverage. This tool also allows you to comment on blocks of code which do not need to be covered.


Is Vagrant an operating system by itself?

No, Vagrant is a tool for creating virtual environments, which it then can run inside a virtual machine (originally only using Virtualbox, but nowadays also KVM and VMware among others).

The environment created by Vagrant is, however, a fully virtualized OS (e.g. Ubuntu).


Well, the extra bells and whistles is called process isolation, a container gets its own namespace from the host kernel, that means the program in the container can't try to read kernel memory or eat more RAM than allowed.

It also isolate network stacks, so two process can listen on port 8080 for exemple,
you'll have to handle the routing at host level, there's no magic here, but this allow handling the routing at one place and avoid modifying the process configuration to listen to a free port.

Secondly a chroot is still read/write, any change is permanent, a docker container using aufs will start from a clean filesystem each time you launch the container (changes are kept if you stop/start it IIRC).

So while a container may be thought of as process namespace + chroot, the reality is a little more complex.


Is there a way to know which files have changed in a docker container compared to its base image?

The docker container diff command shows the differences between a container and its base system.

In the output, files are reported as (A) if the file or directory was added, (D) if a file or directory was deleted or (C) if a file or directory was changed. File permission changes are reported as (C).

she$ CID=$(sudo docker run --rm -d ubuntu sleep 500)
sh$ sudo docker exec $CID touch newfile
sh$ sudo docker exec $CID useradd bob
sh$ sudo docker exec $CID chmod o-x /usr/bin/cut
sh$ sudo docker diff $CID
C /etc
C /etc/group
A /etc/group-
C /etc/gshadow
A /etc/gshadow-
C /etc/passwd
A /etc/passwd-
C /etc/shadow
A /etc/shadow-
C /etc/subgid
A /etc/subgid-
C /etc/subuid
A /etc/subuid-
A /newfile
C /usr
C /usr/bin
C /usr/bin/cut
C /var
C /var/log
C /var/log/faillog
C /var/log/lastlog
C /var/log/tallylog


How to arrange builds for staging and production with Azure DevOps?

I think I see what you are after. Build variables created in the Variables tab of the GUI designer are scoped to the entire definition. But you can set a Job Scoped variable from a script within a GUI based pipeline.

So within the context of each of the Agent Jobs you have defined above, you would add a powershell task that would allow you to set the variable(s) that you then use for the automagic variable substitution in your xml.

YAML is better

If you want to go the YAML definition route you can actually create Job scoped variables directly in yaml. This example below substitutes against a JSON file but the concepts are similar for xml.

This is the Json file it substitutes against test.json.

{
    "menu": {
        "id": "file",
        "value": "File"
    }
}

trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

jobs:
- job: Production
  variables:
    - name: menu.id # Substituting this value in test.json
      value: 'productionValue'
  steps:
    - task: FileTransform@1
      displayName: 'File Transform:'
      inputs:
        folderPath: '$(System.DefaultWorkingDirectory)'
        fileType: json
        targetFiles: test.json
    - task: PowerShell@2
      inputs:
          targetType: 'inline'
          script: |
            Get-Content -Path test.json

- job: Staging
  variables:
    - name: menu.id  # Substituting this value in test.json
      value: 'stagingValue'
  steps:
    - task: FileTransform@1
      displayName: 'File Transform:'
      inputs:
        folderPath: '$(System.DefaultWorkingDirectory)'
        fileType: json
        targetFiles: test.json
    - task: PowerShell@2
      inputs:
          targetType: 'inline'
          script: |
            Get-Content -Path test.json

Custom Conditions

Another thing you might want to dig into is that Azure DevOps exposes task level custom conditions. You could add both transform tasks for Stage and Production to the pipeline definition and then only conditionally run the step you want. You could potentially collapse this into just one job and only build what you need to.

You need to decide on what that condition is. For example, you could do it based on the source branch (probably better) or just a variable that you can set at queue time.


Mitigating Docker Compliance Issues - Can I put everything in the `daemon.json`?

Check out the dockerd options.  This will apply to all containers managed by the Docker Daemon.

Docker also has quite a bit of documentation on how to enforce different standards and compliance here.


Why do many pods fail to communicate with the master

Try to remove the easyrsa relation with kubernetes-master in juju.

juju remove-relation easyrsa kubernetes-master

Recreate it afterwards:

juju add-relation easyrsa kubernetes-master

The certificate seems to be recreated for the master, which should now include it's clusterip (10.152.183.1) address as well in the SAN record of the certificate.


Since the response is Access denied for user so the port and the ip address is correct.

I guess the only issue as @James said  will be in the permission, you have to allow root access to mysql from elk and php containers, you can do it in easy way but it's not secure:

GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY 'password';

%: mean match all

or you can specify who has the access per domain name or ip:

GRANT ALL PRIVILEGES ON *.* TO 'root'@'172.18.0.2' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON *.* TO 'root'@'172.18.0.3' IDENTIFIED BY 'password';

PS: don't forget to change the password to match you root password


If you want to create your own IAM policy, you can use the visual editor present in IAM. Using visual editor, you can construct a new policy from scratch using the very intuitive GUI.

Goto IAM, then policies, then use visual editor.


Where may I need Docker Compose as Kubernetes won the container orchestration war?

Docker compose is still a comparatively easy and light-weight tool to run multiple docker containers with service discovery.
Kubernetes is a metaphorical elephant.  Great if you need to smash Roman lines and win the battle, but overkill for a minor skirmish, and definitely more trouble than it's worth if you're trying to fight a small battle in the mountains.
If you have a fairly large app with multiple microservices, then Kubernetes is worth every penny.
But when you're trying to run just a couple of containers, especially if it's only on a single host (i.e. your frontend + backend and maybe a local instance of RabbitMQ), docker-compose is a much simpler/faster way to orchestrate it.  Kubernetes is like using a cannon to kill a mosquito when a flyswatter will do.
Example use cases where docker-compose works just fine:

Local development environments running on your machine.  Even if you run Kubernetes in production, there's likely a lot of orchestration around getting it to work which would make it too complex for a single machine.
Simple apps without a lot of components
Single tenant apps which run on a single host
Deploying docker-based apps on bare VMs or physical hosts


How to store credentials that are required by an application?

Proper management of an application's secrets has always been a challenge. New challenges came with the adoption of the cloud. There's a great OWASP presentation about the reality and challenges of storing secrets in the cloud.

You might be surprised to hear that storing secrets into the source code is one of the solution (or "architecture") presented. That's because, right now, there is no perfect architecture or way of doing this. In the end, your secrets might be encrypted... but what is guarding the encryption key? "Turtles all the way down", they said.

Every type of secret management has its strengths and weaknesses and the presentation already covers that. Instead, I'll try to go over some of the features you might be looking for in an secret (credential) management solution:

Access control: can you give write access to admins and read access to applications? Can you limit what applications are able to read (application A has access only to those secrets)?
Audit logs: required for many compliancy reports and a good way to figure if something is awry
Secure storage of the secrets: how is the solution storing the secrets? Encrypted DB? Encrypted FS? Who/what holds the encryption key, if any? How is this key used - once at startup and then securely discarded?
Keys/passwords rotation or renewal: if a secret is compromised, can you revoke it and send an updated secret to the applications? Can/should the applications pool the secret management service?
Compatibility: Some of these solutions offer tight integration with certain languages or framework. Some offer REST API. Are you interested in this?

By looking at these items, how they are important to you and how they are implemented by the solution, you will be able to choose one of the secret management service out there.


Reset Gitlab Project to Repository Content

The solution I found in the end is to just create a blank new project, add that to your git remotes and then push to it.
You will have to manually carry over your Issues and probably some other stuff as well. Note that issues can be exported and imported as well, but they loose their comments in the process.


What's better: 2 pods (on the same server) or 1 standalone application?

Is the app stateless

The software is stable as such but my concern is pods, by nature, are meant to be replaceable

This is your primary concern. Is the app stateless (e.g. can the requests be served by any of multiple instances without problem?). If not - be very careful to run it on Kubernetes (e.g. don't use rolling-deployment) and don't run the app in the cloud :)
Apps running on Kubernetes or cloud should follow The Twelve Factor principles
Number of Servers
It would be good to be able to use two or more servers. This reduce the Single-Point-Of-Failure, and you can e.g. take down one server for maintenance but still serve your users.
App directly on Server or in Container

pods (on the same server) or 1 standalone application?

An application runs as a process on the Operating System, whether you run it on a container or as non-container process. There is almost no runtime overhead, what you get with containers is repeatability and maintainability.
A container is designed to be hermetically sealed, so it is not sensitive to your servers configuration e.g. what language-settings or character encoding. This was a bigger problem before containers was used. With containers the app runs in an identical environment every time, except the kernel. In addition it is easier to maintain and manage processes when run as containers, you can e.g. set limits for CPU usage, Memory limit and they get its own network e.g. multiple containers on the same server can listen to port 8080 (from the applications standpoint).
If your apps run as containers on a server cluster, you can also easyli scale out or in by adding or removing servers.
Control of resources

My guess is that once 1 pod gets into to much trouble (memory pressure to name a thing) Kubernetes is going to restart it, as pods are meant to be replaceable.

Yeah, but this is a good thing. Kubernetes is mostly self-healing. If you don't have such "control-loops", you will need manual internvention - since it is the same app that e.g. may have memory leaks.
Run app on Kubernetes or not

whether we do or don't use Kubernetes doesn't depend on this part of the implementation alone

The first thing to check: Is the app stateless?
and also, you should use two or more servers otherwise you get little advantage of running it in Kubernetes.
There is a cost to maintain Kubernetes. You need skills and you need to maintain an addition system. Make sure it adds enough value for you.


Kubernetes serve static content with ingress

You can't configure your Ingress to serve static data (from your host files for example) alone because the Ingress resource is just a way to configure a way of understanding for Kubernetes on how to access a specific internal resource from outside the cluster.

An API object that manages external access to the services in a cluster, typically HTTP.
Ingress may provide load balancing, SSL termination and name-based virtual hosting.
-- Ingress | Kubernetes - https://kubernetes.io/docs/concepts/services-networking/ingress/

You must configure a web server (NGINX, Apache, etc.) with a mounted volume bound to your files. You'll also need a Service resource to publish the web server access internally. Then the Ingress resource will do the work.


Yes.  Check out this example for a nice tutorial to get started.
Per the StorageOS documentation you shouldn't even have to worry about this.  Just let StorageOS handle the namespaces for you with one StorageOS install.


What options are available to encrypt the content of git repositories?

I personally am a fan of Mozilla SOps, we've introduced it to two teams and it's gone down well both times, it has proven easy to implement both with local development and our CI/CD pipelines.

It has support for PGP/GPG to encrypt secrets but also supports Amazon KMS, GCP KMS and Azure KeyVault thus works well in an enterprise cloud-native infrastructure.


Curl Error in HTTP2 framing layer when trying to install Jenkins plugin

If HTTP/2 is not working try forcing HTTP 1.1:
COPY plugins.txt /usr/share/jenkins/ref/

ENV CURL_OPTIONS -sSfL --http1.1
ENV CURL_CONNECTION_TIMEOUT 60
RUN /usr/local/bin/install-plugins.sh < /usr/share/jenkins/ref/plugins.txt


Are microservices "cloud-ish medium-lived services"?

Are those typically "microservices" or is this something different?

"Cloud-ish" services could include microservices. I would include services that are replicated and/or load-balanced, such that you can sustain downtime on a single machine, VM, or running binary without service disruption.

Their differentiation of VM-like services looks like services which have a lot of local state (in memory or on disk) where frequent restarting would incur lots of downtime (from either the time it takes to boot/populate a new VM or because the service is singly-homed.)

  Would you put applications that scale on demand as part of those
  "cloud-ish" services, or would they fall into the "lightweight
  compute engine" category instead?

They could be in either category. Scaling on demand is a commonly-used "feature" of Cloud computing. Think of "Lightweight compute engine" tasks as things like batch jobs, manually run jobs, one-offs etc.


How to add LinkedIn's Burrow to a Docker Compose?

This is the example I was looking for: https://github.com/linkedin/Burrow/blob/master/docker-compose.yml


Jenkins Pipeline Kubernetes Agent shared Volumes

To solve this, you need to include the volume mounts in the containers described here:

Sample Jenkinsfile:

pipeline {
    agent {
    kubernetes {
      label 'test-pod'
      defaultContainer 'jnlp'
      yaml """
apiVersion: v1
kind: Pod
spec:
  containers:
  - name: frontend-test
    image: centos:7
    command:
    - cat
    tty: true
    volumeMounts:
    - mountPath: '/opt/app/shared'
      name: sharedvolume
  - name: backend-test
    image: centos:7
    command:
    - cat
    tty: true
    volumeMounts:
    - mountPath: '/opt/app/shared'
      name: sharedvolume
  volumes:
  - name: sharedvolume
    emptyDir: {}
"""
        }
    }
    stages {
        stage('Test Pipeline Configuration') {
            steps {
                container('frontend-test') {
                    sh 'touch /opt/app/shared/test_file'
                }
                container('backend-test') {
                    sh 'ls /opt/app/shared/test_file ; echo $?'
                }
            }
        }
    }
}

Result:

This solved the problem.

[Pipeline] podTemplate
[Pipeline] {
[Pipeline] node
Still waiting to schedule task
All nodes of label ‘test-pod’ are offline
Agent test-pod-wv12q-brfxj is provisioned from template Kubernetes Pod Template
Agent specification [Kubernetes Pod Template] (test-pod):

Running on test-pod-wv12q-brfxj in /home/jenkins/workspace/pipeline-test
[Pipeline] {
[Pipeline] container
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Test Pipeline Configuration)
[Pipeline] container
[Pipeline] {
[Pipeline] sh
[pipeline-test] Running shell script
+ touch /opt/app/shared/test_file
[Pipeline] }
[Pipeline] // container
[Pipeline] container
[Pipeline] {
[Pipeline] sh
[pipeline-test] Running shell script
+ ls /opt/app/shared/test_file
/opt/app/shared/test_file
+ echo 0
0
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] End of Pipeline
Finished: SUCCESS


GitHub plugin doesn't pass payload data to any variables available from inside the pipeline.

Both Jenkins tracker and GitHub plugin project have a number of open and cancelled feature requests requesting this, for instance, https://issues.jenkins-ci.org/browse/JENKINS-24291.


How do I set up certbot, with Nginx, both in a Docker container?

Answered my own question, documented my steps on my Github repo.

Recap:

Add Nginx config to desired domains:

location ~ /.well-known/acme-challenge {
        allow all;
        try_files $uri $uri/ /index.php;
    }

Use certbot staging to try out test certificates before running the real deal.

sudo docker run -it --rm -v /some/place/to/save/letsencrypt:/etc/letsencrypt -v /some/place/to/save/lib:/var/lib/letsencrypt -v /some/place/to/have/html:/data/letsencrypt certbot/certbot certonly --webroot --register-unsafely-without-email --agree-tos --webroot-path=/data/letsencrypt --staging -d example.com

Run the real thing once ready

sudo docker run -it --rm --v /some/place/to/save/letsencrypt:/etc/letsencrypt -v /some/place/to/save/lib:/var/lib/letsencrypt -v /some/place/to/have/html:/data/letsencrypt certbot/certbot certonly --webroot --email someemail@account.com

And then add in Nginx configs redirecting from http to https with ssl cert locations, updating nginx in docker-compose.yml to have access to certs. Go a tiny bit more in-depth about it in my repo


What topics should at least be included in a Java for DevOps course?

Even with the rationale you edited in, I don't really see a point.

DevOps is an intersection of Dev, Ops (and more, like Security and maybe Testing) in a cultural sense. But on a technological sense, you will always have people in a mixed team who are good at one of those but not the others, and that is fine.

For example: you might deploy something like Kubernetes or OpenShift; and then you will for sure have ops-sided people who are expert at running the low-level stuff, tuning your NAS, adapting your orchestration to a cloud provider of your choice, or to on-premise bare-metal servers and so on. You will not, usually, have all of your developers do the same; they may use those things provided by the others. The ops guys might never see a piece of application code at all; and you would still call it "DevOps".

What your devs will be able to do is manage the full lifecycle of their applications, from source code to production, using the tools provided by the ops guys. When one says that devs do jobs that ops used to do in the past, one means that the automation makes it so that the environments (dev, test, int, prod...) are mostly the same; that deploying to them is as automated as humanly possible (without requiring manual ops, and most importantly without requiring a handover of binaries etc. between devs and ops); that there is a complete, unbroken technological chain from source to prod; including things like virtual networks automatically springing into existence and so on. One does not mean that devs troubleshoot hardware or networking on a low layer.

The same goes for your ops guys. They will most certainly not try to change the application code proper after a few seminars on it, unless you are talking about really small companies with 3-person teams who need to do so out of sheer necessity.

Note that I am not saying that it is not possible to be one person who does both development and operations work. Is is an absolute gold mine to have such people available; and some companies foster that by, for example, rotating devs into low-level ops groups occasionally. But those guys are not your target audience. By the topics you listed in your question, you are targeting ops guys who have never in their life coded anything ("hello world").

If you actually want your ops guys to grow into coding instead of just clicking around in GUIs, modifying or creating config files and such, then here are some suggestions:

If they are writing the occasional bash script already, point out advances constructs to them (loops, functions, error handling, etc.).
Try to get them interested in some dynamic language like perl or ruby for a low-hanging fruit, one step above shell scripts.
Introduce them to go, which is really nice for standalone command line applications and especially automation tools. It should satisfy the need to learn something new which should go really well on anyone's resume, and will satisfy the naysayers regarding non-natively compiled languages.

... and so on. But don't try to sell them "Java" as a route into "DevOps".


Copy multiple files using * wildcard from kubernetes container

Wildcard support was added to kubectl cp in the #72641 pull request.
It is merged but only available in Kubernetes v1.14.0. Take a look at the ChangeLog:

Notable Features:
[...]
• kubectl supports copying files with wild card
(#72641, @dixudx)

However, as stated on the issue #78854 the PR was made without proper tests and the feature is actually broken. The PR to fix the problem (#78928) is open since June, and it's not merged yet.
Although the PR to fix the feature exists, they are also considering removing the support to wildcard copy.


The swarm protocol is not http, so you will not be able to pass it through a proxy. Swarm nodes need direct communication between each node on port 2377/tcp (or your configured swarm port), 7946/both, and 4789/udp. If you configure an overlay network with the secure option for IPSec, then you'll also need protocol 50 open between the nodes.


docker volumes on cifs share cause permission disaster

The problem can be solved by using mount options that force the application of the correct user and group eventhough these attributes can't really be set on the target system its sufficient to get around the docker related problem.

Also a reasonable set of privileges needs to be defined that is used for new files and folders.

mount -t cifs -o username=user,password=pass, \               # provide creds
  uid=www-data,forceuid,gid=root,forcegid, \                  # force gid and uid
  file_mode=744,dir_mode=744 //host/share /local/mountpoint   # set permissions for new files and directories


Is there an image on dockerhub that is able to run make?

GNU make can be found in the gcc container.

  Compile your app inside the Docker container
  There may be occasions where it is not appropriate to run your app inside a container. To compile, but not run your app inside the Docker instance, you can write something like:

  $ docker run --rm -v "$PWD":/usr/src/myapp -w /usr/src/myapp gcc:4.9 gcc -o myapp myapp.c

  This will add your current directory, as a volume, to the container, set the working directory to the volume, and run the command gcc -o myapp myapp.c. This tells gcc to compile the code in myapp.c and output the executable to myapp. Alternatively, if you have a Makefile, you can instead run the make command inside your container


Automated docker builds with private bitbucket server

I would use a Packer template, specifically with a Docker builder to do this. Triggering this build can be done with hooks (either BitBucket webhooks, or git server post-receive hooks.

Building a Docker image from a Packer template has some benefits, but requires Docker Engine installed - so, doing the build with a Dockerfile would also be an option.

The tricky part would be the management of credentials. Building the image does not require credentials, nor does tagging -- however you need to be authenticated to push the image to the registry. If the build happens on the same server that is hosting BitBucket, you could keep them in a protected file and execute the build via the post-receive hook in a shell script. This would be a lightweight option, especially if the server itself is locked down.

So, in summary :

Install Docker Engine on the BitBucket Server
Write a script which can consume registry username and password, to build the image, either from a Dockerfile or from a Packer template.
Add a server-side git hook to execute this script to build, tag and push the image.


Rename VM in VMware Fusion via command line

Well, there is no rename, it is a multi-step process, which involves renaming a bunch of entries inside the vmx file, using vmware-vdiskmanager to rename to vmdk and in general doing a bunch of potentially problematic actions if you don't know what you are doing.

You could probably write a quick Perl script to do all that using this guide.

Another option would be to clone it and delete the original, but it is resource intensive and does way too much work for what it is worth. The code below is untested:

vmrun -T fusion stop "/Volumes/vms/old/old.vmx"
vmrun -T fusion clone "/Volumes/vms/old/old.vmx" "/Volumes/vms/new/new.vmx" full --cloneName=new
vmrun -T fusion deleteVM "/Volumes/vms/old/old.vmx"
vmrun -T fusion start "/Volumes/vms/new/new.vmx"


Change container size in a template from the Marketplace

According to https://azure.microsoft.com/de-de/blog/container-apps-now-available-in-the-azure-marketplace/ it seems to be possible to define the storage-size when a container-app is chosen.


How to create a second IP just for dockers

If you pass the listen address as the very first component of -p, Docker will only listen on that address:
$ docker run -p 127.0.1.2:8080:80 -d nginx`
$ ss -ltp | grep 127.0.1.2
LISTEN  0       4096         127.0.1.2:http-alt          0.0.0.0:*


What is the meaning of source artifacts that must be in a scripted form?

IMO these could be possible synonyms for scripted, in the context of your question here:

in a textual format, which you can edit in some editor, like YAML, XML, JSON, PHP, etc.
NOT in a binary format, the result of some build process (like a .EXE file, etc).

And I bet the reason for the "key prerequisite" (as in your quopted text), is that in the end you want to be able to perform some sort of comparison between 2 versions of such files. E.g. like a patch file (if usig GIT). Think about how a compare of 2 versions of a binary looks like, pretty sure every line is updated (or: all old lines are deleted, and all new lines are inserted).

This applies to pretty much any operating system, be it Linux, Windows, ..., or even zOS (aka good old mainframes).


can't access my instances by public IP in a VPC (aka: what is wrong with my terraform)?

Your default security groups and newly created security groups include default rules that do not enable you to access your instance from the Internet.
Add this:

resource "aws_security_group" "allow-ssh" {
  vpc_id = "${aws_vpc.vpc.id}"

  ingress {
    from_port = 0
    to_port = 22
    protocol = "tcp"
  }
}

Then add the line vpc_security_group_ids = [aws_security_group.allow-ssh.id] to the instance definitions.


When I did echo %USERPROFILE% from Jenkins it returned null. So I had to
manually set USERPROFILE value from Jenkins as environment variable. Also I had to set these

APPDATA=%USERPROFILE%\AppData\Roaming

LOCALAPPDATA=%USERPROFILE%\AppData\Local


If you see the logs, there's exactly 15s between the calls. Which is what you have configured in you docker compose file.
If you don't want it to keep pinging after it finds out that the server is alive, you have two options (which is not recommended to do):

Don't use healthcheck, just add your logic in the command part
Setup a higher interval for your healthcheck

Also, I have questions for you:

Why do you want the healthcheck to be done only once?
What if you server goes down, don't you want to restart it?

My recommendation is to keep this config as is because it does the work of checking for your service availability and will restart as needed and quickly.


How to manage Consul and its quorum in auto-scaling environments?

I would set the leave_on_terminate option to true. As per the documentation

  leave_on_terminate If enabled, when the agent receives a TERM signal,
  it will send a Leave message to the rest of the cluster and gracefully
  leave. The default behavior for this feature varies based on whether
  or not the agent is running as a client or a server (prior to Consul
  0.7 the default value was unconditionally set to false). On agents in client-mode, this defaults to true and for agents in server-mode, this
  defaults to false.

What happens when a node is shutdown gracefully is to send SIGTERM to all processes before shutdown, with this setting on the consul agent will leave the cluster so it won't be considered as a node which can restart and be back in the cluster in a few hours (which is what your quote says it does by default).


How to add multiple jvm options in a saltstack jinja?

Perhaps a macro might be what you are looking for.

Understanding Jinja:Macros

In the lib.sls example in the link (pasted below), it sets a value contingent on the value of the if statement.  I think a similar approach could work for you.

{% macro pythonpkg(pkg) -%}
  {%- if grains['os'] == 'FreeBSD' -%}
    py27-{{ pkg }}
  {%- elif grains['os'] == 'Debian' -%}
    python-{{ pkg }}
  {%- endif -%}
{%- endmacro %}


Accessing AWS Secrets Manager from Docker

Access to AWS Secrets Manager requires AWS credentials. Those
  credentials must have permissions to access the AWS resources that you
  want to access, such as your Secrets Manager secrets. The following
  sections provide details on how you can use AWS Identity and Access
  Management (IAM) policies to help secure access to your secrets and
  control who can access and administer them.

so you need to provide the aws credentials to docker which has access to the manger

you can follow on of these methods:

providing the credentials with run command:

docker run -e AWS_ACCESS_KEY_ID=XXXX -e AWS_SECRET_ACCESS_KEY=XXXX myimage

or providing the file ~/.aws/credentials and then create your container:

docker-machine create --driver amazonec2 --amazonec2-open-port 8000 --amazonec2-region us-west-1 aws-sandbox

see this


Docker container and their IPs on overlay (swarm) networks

When you ping nginx the docker swarm returns Virual IP of swarm service named "nginx" not the container ip.

This is is how internal load balancer works - you can have multiple replicas of "nginx" containers behind the "nginx" service.


What are the implications for Docker Community Edition of the Docker acquisition by Mirantis?

I'd ask the question the other way around, rather than will Docker CE continue to share the Docker Enterprise codebase, it's whether Docker Enterprise will continue to share the upstream Docker CE / Moby codebase. They've always maintained an internal closed source fork for the Enterprise code, which has additional features. However, the development process of CE has always been open source, and features added to CE were mirrored from CE to the Enterprise codebase. The acquisition of Enterprise didn't change that it was already a fork of CE. And the acquisition did not include the CE code or any of the other open source projects, those continue to be managed by Docker, which they will maintain as a core component to their business (CE is embedded in the various Desktop products, and Hub has little value without it).

The biggest downside I see to the acquisition is the loss of staff from Docker to Mirantis. I'm sure many people had a dual role working on Enterprise and CE, and a significant chunk of the staff went to Mirantis. That said, the CE repos are open source, Mirantis staff can submit PR's for changes they'd like usptream, and they can pull down new releases of CE into the Enterprise product. The end result for users may end up looking identical to what we have today, just two close companies coordinating, rather than two sides of the same company. From an open source perspective, having more outside contributions is actually a good thing, since it forces governance and planning into the open, that in the past may have happened with internal meetings.

Disclaimer: I have been exposed to private discussions from the Captains program on this topic (not much since they are still working this out) and have done my best to keep my answer to only the public knowledge.


Does building a Docker image require the Docker daemon to be running?

Each command in the Dockerfile is processed as a separate step. Every step from the build process gets executed in a dedicated container. It starts from Step 1 (the first command in the Dockerfile). That step is run in a container and if it is successful, a commit to a temporary image (let's call it image A) takes place. Then, Step 2 is executed in a container which is run from image A ... and so on until the final step is executed successfully. And as Tensibai commented - how do you think to run a container without the daemon?

So, yes - building a docker image requires the docker daemon to be running.


Azure DevOps Pipelines running on incorrect branches

I have solved this issue, it had to do with the path to the pipeline.
I created a directory called pipelines/dev and pipelines/prod in the root of my repo. And I pointed either build pipeline to the respective yaml config file. Now it will only build on the commit of the specific branches.


Has `docker-compose` to be phased out in favor of `docker stack`?

There are a few features of docker-compose that are not implemented in docker stack deploy and vice versa.

With docker-compose, you can run privileged containers and change capabilities which is not supported with swarm services yet. You can also run a container that is designed to exit, e.g. a batch job, more intuitively in docker-compose than in swarm mode. If your containers are designed for sysadmin tasks, this may make more sense.

With docker stack deploy, you get access to secrets and configs, along with replication settings, configuration of rolling updates, ingress networking with the routing mesh, and various other swarm mode features. If your containers are designed for highly available microservices, this makes more sense.

Both of these use the same docker-compose.yml definition file (with version 3 required for swarm mode). That makes moving between the two fairly easy. There are also projects like compose-on-kubernetes and kompose that make the transition to kubernetes with the same stack definitions easy.


is a secrets-provider-container a good idea?

There is probably a better way if you are working with Google Cloud Run.  I have not tried this but I see that they have "Secret Manager" that you can use.  They also mention a project Berglas that can be used to store secrets in GCS cloud storage.
In my personal experience, it find it best to leverage widely adopted solutions and avoid creating a unique solution for security.  Getting it right is non trivial and it is less code you have to support.
This FAQ has doc links to the solutions I mentioned.  https://github.com/ahmetb/cloud-run-faq#how-to-configure-secrets-for-cloud-run-applications


What is the biggest Docker image on Docker Hub and how to find it?

If you're looking for a master list of all images and sizes, kind of a 'statistics' pages, Docker Hub doesn't have that. You'll need to use their HTTP API to:

get all the repositories, stepping through the paginated results. API reference: https://docs.docker.com/registry/spec/api/#listing-repositories
pull the Image Manifest for each repo. API reference: https://docs.docker.com/registry/spec/api/#pulling-an-image-manifest
Store those manifests in local database so you can sort them based on image size

You'll specifically need to pull the newest Image Manifest Version 2, Schema 2, which apparently not all repos support yet. This newer manifest has the image sizes in it: https://docs.docker.com/registry/spec/manifest-v2-2/

All in all, this is much faster than trying to actually download all the images from Docker Hub, but it will still take quite a long time and a lot of processing. The database for storing this all might get rather large and difficult to process, so I'm not sure how much you really want to pursue this route for a simple research project. That's up to you.

You may want to take @Vish's advice and simply go for obvious large images such as the Microsoft or Java ones.


Weird requests filling apache access logs

You're looking at a hex-escaped string.  After some googling, I found this entry on server-fault:  https://serverfault.com/a/758511/90491.

You're looking at a non-decrypted HTTPS request.

From Link:

  You are most likely seeing this because you are making an HTTPS request to an HTTP endpoint. For example, you're sending an HTTPS request to port 80 of your web server instead of 443. As a result, the HTTP endpoint gets a bunch of encrypted data, makes no effort to decrypt it (since HTTP is supposed to be plaintext), and so you get a bunch of gibberish in the log file.


Why does Docker define invalid IP address of service?

Ok, I have found the reason:

I tested starting the services with different environment variables for db. And after that I shut the services down, changed the db name, username and password, and started the services again.

But I forgot that db folder /var/lib/mysql is mounted in a volume, and it remembered my previous data; everything was mixed up. So it's not problem in network, it's mixed data in mysql (different passwords for the same user).


I don't think what you want to do is possible.

I suggest you to use a managed compute environment so you don't care about cluster resources. Set the min and max CPU for the compute environment (min 0 is recommended so you don't waste many when no jobs are running). Then only set the resources needed by your jobs and batch will add compute capacity to satisfy the demand.

Your costs will be the same because batch will create an ASG(autoscaling group) and ECS cluster but it only charges you for EC2 instances.


Multi-container server of PHP apps with reverse proxy?

Problem is in resources - I see that each app container has own nginx instance, and server (VPS) must have global nginx, working as reverse proxy for containers.

Generally speaking yes, this is how this sort of thing goes.  It's more of an issue for PHP than other languages because PHP doesn't really have a standalone application server, but instead is commonly deployed using php-fpm with nginx.  Since php-fpm supports communication via TCP sockets, theoretically you could run only fpm in the containers and have nginx on the host talk to fpm across the Docker networking layer, but I haven't researched this option enough to be convinced it would work.

Docker makes the most sense when used within the context of an orchestration layer like Kubernetes.  Rather than running several Docker containers manually or semi-manually on a particular host, you would schedule them with kubernetes and it would distribute them across its cluster of nodes - the specific machine that runs the container becomes irrelevant.  The kubernetes service then provides you with an IP via kube-proxy that load-balances across the pods of that service.


Jenkins Declarative pipeline: Execute stage conditional on execution of previous stage

A pretty simple solution is to declare a global variable on top of your pipeline, and check it's value in your last conditional stage:

def flag = false;

pipeline {
    stages {
        stage('1 - Always') {
            steps {
                sh './always.sh'
            }
        }

        stage('2 - Maybe') {
            when { condition }

            steps {
                sh './maybe.sh'
                script { flag = true }
            }
        }

        stage('3 - If Maybe was executed') {
            when { expression { flag == true } }

            steps {
                sh './conditional.sh'
            }
        }
    }
}


Looking for good books on Immutable Infrastructure

Although I am willing to bet that this answer may not age well, I would suggest

"Infrastructure as Code" by Kief Morris for the "theory" part.

For the "application" part, I would suggest "Terraform up and Running" by By Yevgeniy Brikman.


Is it possible to pass environment variables to multibranch pipeline job in a time of generating it via DSL?

You can use Jenkins DSL and pass the environment variable as mentioned in below links:

https://jenkins.io/doc/book/pipeline/syntax/#environment
https://jenkins.io/doc/pipeline/steps/workflow-basic-steps/#withenv-set-environment-variables


Modify sudoers file with ansible playbook template

TL;DR: KISS. Don't use less.

People often make a mistake with ansible by trying to make variable things that don't need to be. Unless there are multiple places where you define the list of commands that support can access, it is perfectly acceptable to just put them in the template create file:

templates/etc/sudoers.d/support1

Cmnd_Alias LS = /bin/ls
Cmnd_Alias LESS = /bin/cat
Cmnd_Alias DU = /usr/bin/du

%support1 ALL=(ALL) NOPASSWD: LS, LESS, DU

or even explicitly as you don't reuse the Cmnd_Alias anywhere

%support1 ALL=(ALL) NOPASSWD: /bin/ls
%support1 ALL=(ALL) NOPASSWD: /bin/cat
%support1 ALL=(ALL) NOPASSWD: /usr/bin/du

And add some task like:

- name: add templates
  template:
    src: {{ item }}
    dest: /{{ item }}
    owner: root
    group: root
    mode: 0640
  with_items:
    - etc/sudoers.d/support1

You would only use templates instead of files because late on there might be some variable to add to all those or the group name might come from variable if you get another role that creates the groups.

If you need to use variable, the thing you can do is to use a list of hashes like this:

sudoers.support1.commands:
- { alias: "LS", path: "/bin/ls" }
- { alias: "DU", path: "/usr/bin/du" }

Then in the template:

{% for item in sudoers.{{ group }}.commands %}
Cmnd_Alias {{ item.alias }} = {{ item.path }}
{% endfor %}
%{{ group }} ALL=(ALL) NOPASSWD: {{ sudoers.{{ group }}.commands | map(attribute='alias') | join(', ') }}

It is not safe to use /usr/bin/less

In all this you did not notice much important thing and that is the use of less as viewer. Sadly that is a security hole. You can type '!bash' for invoking bash. And by pressing 'v' you get into editor based on VISUAL, EDITOR or LESSEDIT variables. So you can give them '/bin/cat' and they can always pipe the content into less themselves. Note this is still a security hole as some files in unix are very intentionally restricted for example:

/etc/shadow
/etc/sudoers
/etc/ssh/ssh_host_rsa_key
$HOME/.ssh/id_rsa


Import data from SharePoint to Azure DevOps

There are no tools to import data from SharePoint to Azure DevOps. You can consider the following:

Export SharePoint List to csv and then import it to Azure DevOps
Use Excel integration. Export Sharepoint list to Excel and then copy/paste data to Excel linked with Azure DevOps and publish it.


Docker exec environment different to container shell

Please refer to bash initialization process.
https://www.gnu.org/software/bash/manual/html_node/Bash-Startup-Files.html

In your Dockerfile, at lines 12-14 you are patching files, but do not activate any of rbenvs.
Later at your last RUN call, you finally ARE activating an rbenv. This starts pointing at that specific version of ruby (I thin this patches root files)
At that point you have different global shell initialization files vs. user initialization files.

Difference

entry point: "non-login" initialization process happens, then the file: /root/.bashrc is executed (a.k.a. "sourced"), it points to your cool ruby
exec of ruby: ruby executable is located using your default PATH. without running /root/.bash* files.

To validate all this you can append printouts to your shell's initialization files, and see what gets printed upon 2 cases.

Regards.


You have this in your config file:

[inventory]
enable_plugins = host_list, script, auto, yaml, ini, toml

Do you really need all these plugins?  If not, they'll just add time to your processing.  There is also a small chance that one of them is successfully parsing your inventory file as an empty inventory.  And the documentation states that

  Once an inventory plugin succeeds at parsing a source, any remaining inventory plugins will be skipped for that source.

If you do not need those plugins, I'd comment that line (or rework it to remove some of the items).

Also, are you sure that it is using the local hosts.ini?  The command ansible-config might help confirming that.

Regarding your first question ("Why I have empty hosts.ini in my execute directory"), there is nothing in your playbooks pointing to that file being created or modified.  Did you mean to  have it modified?  Or you don't know how it was created?  The question there is not clear.


How to investigate a main process that has died in a docker container?

General ways to track why a process in Linux failed are good. One such way is to run a process using strace which will tell you the system calls process did and usually point to the reason for a failure.

You can create a Dockerfile that looks something like this:

FROM original_image

RUN apt-get -y update && apt-get install -y strace

# build with `docker build -t debug_version`

Then run your new image using docker run debug_version strace original_cmd.

For processes that fork children (and then die) you want to run strace with the -ff option. You can also map some file using Docker data volumes and use the -o option from strace to write to it. But in general strace will leave output on stdout, which is readable using docker log.

Related Q: Linux process terminates mysteriously


From my experience (where I currently work and another company where a friend works at) the DevOps team's report to the CIO.

Also to note that our DevOps team works a lot with, mostly alongside the CTO on a "day to day" basis compared to the CIO.


Issue with cloudformation in target groups and ASG

As the error message states, TargetGroupARNs value should be of type List. https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-as-group.html#cfn-as-group-targetgrouparns


Problem with logging in docker when using fluentd for logging of three flask apps running in a docker network behind an nginx

I finally found out the problem: the fluent container needs some time to start up; because it's started with --detach the subsequent containers try to connect before fluentd is actually running, only the 3rd instance of the flask application connects when fluentd is already up. So, delaying the startup of the remaining containers solves my problem.


I really hate this kind of discussion, as if there's a mold that fits a good CTO or CIO.

Let's examine the qualities of a good executive rather:

Practices enabling and empowering engineers and local team leadership
Deflects bureaucracy from external sources
Helps remove organizational blockers to change and improvement
Does not mandate architecture from the top down
Acts as an evangelist for software first solutions
Promotes automation and solid foundations for solving problems rather than implementing patches
Knows how to avoid unrealistic deadlines and demands quality, but not perfection, over quantity of features
Uses data to solve problems and constantly evaluates the integrity of the data gathering
Values iterative development rather than long, cumbersome planning processes
Values education and internal training & advancement over outsourcing

I'll add some more if I can think of them. We need to stop assuming a CTO or CIO is a list of resume qualifiers and start hiring people who really understand the process of building, maintaining, and delivering software from sun rise to sun set.


s3fs to mount S3 bucket with iamrole on non-aws machine

Have you got AWS credentials configured on that machine?

~ $ aws configure

And then verify with:

~ $ aws sts get-caller-identity

Hope that helps :)


CloudFormation is service that allows you to create your infrastructure from code to automate your deployments.

You can set up a rule to run an AWS Lambda function on a schedule by using CloudWatch event.
https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/RunLambdaSchedule.html


Deploying to VSphere with Packer and/or Terraform?

I'm using vagrant with vsphere plugin, starting from a 'base' image already present on vsphere.

The new machine is then fully configured (inlcluding middleware) by chef after vsphere customization.

That's one in many possibilities, unsure this question lead to actionnable answers out of everyone 'workaround'.


Terraform's local state (the 2 tfstate files) contain very sensitive info

Do not store your state in anything besides remote backend. Even if you store it in a remote backend the secrets will appear in a plain text again, this is an ongoing Terraform limitation.

In our case we are using AWS S3 backend ( although we have multi cloud environments ), we've got those key benefits out of the box:

Encryption in transit and at rest.
Access policy configuration.
Durability and availability close to the maximum.
Locking capability.
Versioning.

Please do not go with locally-maintained repos that are redundant as possible.


Log monitoring/metrics dashboard with images/video

In terms of embedding in an external page, you have some options out of the box with both Kibana (docs here) and Grafana (docs here).
If I understand correctly, you are trying to generate realistic clickable thumbnails - in that case, you should try Grafana first since it better supports server-side image rendering


Staging - Test multiple microservice versions together

Yes, this can be done by Service Discovery. The product family you're looking for is of products such as Consul. You should have the container services report their identities and versions when they load. Report their version and ip (for example), and have the tests run against these values.

This will require some effort with installations and scripting, and has a learning curve - but this is the way to go.


Is Docker compose installation required everytime I start a new session on docker CLI

When installing docker on windows or mac, there is a virtual machine called 'docker machine' that runs Linux OS. Docker has a command docker-machine used to start this virtual machine using docker-machine start. It is likely that every time you restart the toolbox, you start a new VM with new fresh copy of the OS. If you do docker-machine ls you should see multiple docker machines, one of them running. If instead on restart you always start the same docker-machine, it will have the previous changes, like previously installed packages, including the docker compose. You can give docker-machine start <arg> with the name of the docker machine you want to start, which should be always the same one.


Git seems to recognize http.sslcainfo property from .gitconfig but ignores it on execution? SEC_E_UNTRUSTED_ROOT

Been hit by this just today after a fresh install so here's how I got over it:

From your logs (emphasis is mine):

  fatal: unable to access ... : schannel: next InitializeSecurityContext failed:

git is configured to use schannel (windows native implementation), but schannel use windows certs bundle and not a the cainfo bundle.

To switch to openssl to use a custom bundle file use this:

git config --system http.sslbackend openssl

And now git will honor the CA bundle passed in http.sslcainfo.

Alternatively as my error was a problem of cypher suite you may be interested by the following links:

https://support.microsoft.com/en-us/help/3161639
in case you have still an old system (Win2k3) https://support.microsoft.com/en-us/help/948963/an-update-is-available-to-add-support-for-the-tls-rsa-with-aes-128-cbc


node.js runs via a command line (no windows involved). If you want to run more than one copy of the same program you can simply start it more that once.
Take this simple one:
https://nodejs.org/en/docs/guides/getting-started-guide/
Since it runs on a dedicated port (3000) if you start it more that once it will fail.
Here is something I found on google that details all the ways to do this:
https://itnext.io/4-solutions-to-run-multiple-node-js-or-npm-commands-simultaneously-9edaa6215a93


Different types of leadership styles are required. To start with DevOps in a company, one has to apply a different leadership style compared to when it has already been implemented.

When one starts with DevOps a cultural change has to be done and engineers should get rid of, for example, the "throw it over the wall" mindset. When DevOps has been implemented, a leader should be able to communicate well with other teams to connect multiple departments in a company.

In summary, in the beginning a leadership style that motivates people to implement DevOps is required. Later communication between teams and how the team will work in the organization, i.e. more focus on external.

A leadership style that could be used to implement DevOps is a transformational leadership style.


if the bottle neck is the number of connections, is it better to scale redis by sharding or by replication?

Redis Cluster could be considered as an OOTB solution for the Scalability requirements. There is another interesting open source library known as Redis Shard for sharding implementations on Redis. Please find the GitHub repository here.


How can I diagnose php cli command execution time inside a docker container

The general process of debugging inside the container is the same as debugging outside of one.  The only thing that you generally need to know is that you can use docker exec to attach a bash shell inside a running container, or override the command with --entrypoint to launch bash instead, and then run the script manually.

At that point, your question becomes "How do I debug slow execution times in a PHP script?".  That is not really in scope for this site, but you can start with a few basic things: add print statements or attach a profiler to see where things are getting slow, and then narrow it down from there.


As far as I can read between the lines, you want your upper management to support you in introducing the 12-factor approach to your company's software development.

In that respect, you do not want to explain each of the 12 factors to them. If they really were interested in that, then you could just send them the 12-factor website you have linked in your question - it already has 1-sentence summaries.

I'll answer the question you wrote in the title:

  How to explain the Twelve Factors to non techy people?

The Twelve Factors are not a soft "culture". It is a set of concrete, practical and pretty low-level techniques for solving problems relating to configuration management, deployment and high availability of applications. Instead of "culture" or "methodology", I would rather call it a pattern, or a kind of handbook for developers. It is obviously related to DevOps, cloud and similar buzzwords, but not necessarily so - you can also employ the techniques if you only have one developer doing his own thing, and even if there is no cloud or otherwise "modern" environment involved at all.

It is not even necessary to involve management in the question whether to employ some or all of the techniques mentioned in the 12-factors. Many of them are simply best practices which everybody who develops ("service"-like) software will end up over the years.  This is pretty evident in the I. Codebase, III. Config, XI. Logs. No manager is required to give a green light to employ these practices.

Other of the principles, namely II. Dependencies, IV. Backing services, VI. Processes, VII. Port binding, VIII. Concurrency, IX. Disposability are neatly combined into the keyword "microservices". Don't explain to your management those specifics, but tell them that you are going to make a move from some presumably monolithic application architecture you had before to a microservice based one. On a management level, this can be sufficiently done with a graphical representation (one big blob of complex software on the one hand - several individual small blobs of software on the other hand). You can explain why the later can be better for the job at hand with nontechnical terms; every manager can understand the concepts of dividing a big, convoluted mass of code into smaller pieces that can be handled (and scaled) much easier.

Your management will want to know what it costs and how long it takes. 12-factor tells you nothing at all about that. While there is a slight cost involved in some of the factors (e.g., establishing a central log stash), the cost is minor and will easily be caught up by savings later. Other costs may be the time needed for your developers to embrace some of the more tougher parts (i.e., II, VII, IX, depending on how much existing software you already have).

Very important: it is pretty simple to work in an environment with a big monolithic application, and only add new stuff with the full 12-factor approach, or convert bits and pieces from the original software as you go. This  might be more sensible if you have time or money constraints.


How to get the next friendly name that Docker will assign to the container it will make?

As the name of the container is randomly generated you cannot get it before it's generated (by definition). You can use this service to generate random names :

NAME=$(curl -s https://frightanic.com/goodies_content/docker-names.php); \
docker run -ti --name $NAME --hostname $NAME ubuntu:latest


Prometheus alert CPUThrottlingHigh raised but monitoring does not show it

The CPUThrottlingHigh is an alert created by the kubernetes-mixin project. There is an open issue (#108) to discuss this alert. I suggest that you read all the comments on this issue to better understand the problem.
In short, the problem is: When working with low CPU limits, spiky workloads can have low averages and still be being throttled.
Also, take a look at this issue (#67577) from Kubernetes project, which addresses a Kernel bug in CFS quotas that may cause unnecessary CPU throttling. The discussion is still open, and the Kubernetes project are even considering disabling CFS quotas for pods in the Guaranteed QoS (see #70585 for reference).
Consider the following options:

Increase (or even remove) your container CPU limits
Disable Kubernetes CFS quotas entirely
(kubelet's flag --cpu-cfs-quota=false)
Use a Kernel version that contains this fix (torvalds/linux 512ac99)


What strategies exist to transition DB Migrations with Stored Procedures to Continuous Delivery?

It is quite difficult to migrate the databases without downtime, however, using some approaches you can do it with very minimum impact, nearly a zero downtime.

The main idea in this is to create new databases with new schema. Then make changes in your code. First migrate the data from the old database to the new database. Then make changes in your code such that it starts writing to both the databases, old and new. You need to decide a time (time when there will be least writes to the database). Then stop all the writes. Remove the old database and make sure all the writes should now happen to new database from now on.
Then start the writes back.

In this way, you can achieve DB migration with minimal downtime.

You can check this for a detailed information on how this can be done.


How to setup Cloudfront for Load balancer for a private staging domain name?

My solution is based on signed cookies (find how to implement it here, https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-signed-cookies.html); I've created a lambda that handles this for me a bit differently, as I need users to access a location to get their cookie in the easiest way possible and have someone control to whom can be granted the auth cookie. An example of a basic implementation of this is here: https://aws.amazon.com/blogs/compute/simply-serverless-using-aws-lambda-to-expose-custom-cookies-with-api-gateway/

Finally, after you have provided your Content origin in Cloudfront, in the Behaviors tab where you setup the Content origin desired behaviour you need to set "Restrict Viewer Access (Use Signed URLs or Signed Cookies)".

Obs: make sure you also forward all cookies and query strings.


Ansible - private git repositories - SSH agent forwarding vs copying private SSH key

You can also consider to encrypt the file containing your private SSH key using Ansible Vault.

Some links:

http://docs.ansible.com/ansible/latest/playbooks_vault.html

https://therealmarv.com/ansible-vault-file-handling/

https://www.calazan.com/how-to-deploy-encrypted-copies-of-your-ssl-keys-and-other-files-with-ansible-and-openssl/


CI/CD pipelines Azure devops automatic merge after deploy release

I would look into Azure Functions. Create a gate trigger and enable azure function after a successful release.

References:
https://docs.microsoft.com/en-us/azure/devops/repos/git/create-pr-status-server-with-azure-functions?view=azure-devops


What options are available to get a value that is associated to a key in yaml?

One could use go-yq if one wants to lookup a value by applying jq style, i.e. if firefox_version then one could look it up by adding a dot:

docker run -v ${PWD}:/ansible-firefox utrecht/go-yq:2.1.0 \
       .firefox_version /ansible-firefox/defaults/main.yml


How to sync two directories on same host using Ansible?

The copy module copies a file from the local or remote machine to a location on the remote machine.

In order to copy the file from the remote machine itself, set remote_src to yes.

- name: Copy multiple Directories
          copy:
            src: "{{ item.src }}"
            dest: "{{ item.dest }}"
            mode: "{{ item.mode }}"
            remote_src: yes
          with_items:
             - { src: '/home_old/',dest: '/home/',mode: preserve}
             - { src: '/var_old/',dest: '/var/',mode: preserve}


Terraform: Is there a concise syntax for specifying multiple tags for a resource?

For an auto-scaling-group this is the most concise syntax available.

For most other resources you use the tags syntax which looks like:

tags {
  Key1 = "value1"
  Key2 = "value2"
}


Vendoring is the concept of downloading/installing a specific version of dependencies and making it available elsewhere (typically, within a local repo/folder of your supporting application)

Vendoring is usually done to prevent breakage occurring when dependences are no longer available. Vendoring also prevents your applications from breaking when dependencies are no longer available( for ex: the leftpad incident)

For example, let's consider you have your Chef cookbook that depends on another third party cookbook, say mysql2 chef gem. Now while mentioning in your metadata.rb, you didn't pin the version or vendor the cookbook. Lets consider the cookbook has had a version upgrade, breaking all previous APIs. The previous versions have also been taken down. In such a case, your cookbooks are suddenly broken as well and if you have a urgent deploy, your Chef run will fail.

Vendoring prevents this by making available the dependency on your local chef server, so even if the dependency has gone ahead, you can continue referring to the previous version and your cookbook will not break

(Also goes without saying, you must keep updating the dependencies and cookbooks, else you might get bit with some bad bugs)


I am not sure about other approaches, but the one that I use is a modification of #3.

There's usually a classification of releases (for example think of Ubuntu LTS and regular release).

For the regular release, I would just release a new version and add a new tag.
For LTS/Stable releases I would delete a tag pointing to a faulty release or change a commit this tag points to


How do you know which secrets and credentials of your production services were used, and by whom/what?

The short answer: you would need to use a secret management system that provides 'audit logs'.

There are many options for handling secrets out there, but unless the tool you use has audit logging, you'll never truly know for sure what credential was used where.

I see you mentioned using Hashicorp's Vault. That's a really good one. They have audit logging built in; they call it 'Audit Devices', which can be enabled with a simple command.

Again, there are many perfectly valid options for handling secrets: Chef vault, 1password, environment variables injected by your build instances, home-grown solution using encrypted files stored on object storage, etc.... However, unless the system specifically says it provides audit logging, you'll never truly know what was used when and where.

You can design a secrets system with the best intentions, and make many rules that employees have to follow in regards to have the secrets are accessed, but for auditing purposes, you want to know where the rules have been broken. If you have a firm requirement to always know when and where secrets where used, I recommend you stick with something like Hashicorp Vault, turn on Audit Devices, and then ship the log somewhere for storage or processing.


What Puppet modules do include best practices regarding CI and rspec-puppet like Geerlingguy's Ansible roles?

Go to https://forge.puppet.com and look for modules with a high "Quality Score"

For example https://forge.puppet.com/puppetlabs/stdlib/scores.

Then you can take a look at the Project URL and view it's .travis.yml and the tests in spec/


When exactly and what for should VOLUME in a Dockerfile be used?

Short version

VOLUME instruction and -v dest are used to create unnamed/anonymous volumes, -v scr:dest option is for mapped volume.

Long Version

Dockerfile's VOLUME does not allow you to specify a host path.
On the host-side, the volumes are created with a very long ID-like name, these volumes are often referred to as unnamed/anonymous volumes.

Given this Dockerfile:

FROM php7:latest
VOLUME /var/www

Build it:

docker build -t myTest

Run it :

docker run --rm -it myTest

Inside the container, run ls and you'll notice the directory exists; /var/www

Running the container also creates a directory on the host-side.

While having the container running, execute docker volume ls on the host machine and you'll see something like this

DRIVER    VOLUME NAME
local     c984..e4fc

Back in the container, execute touch /var/www/myFile

This file is now available on the host machine, in one of the unnamed volumes.

ls /var/lib/docker/volumes/c984..e4fc/_data

Similarly, you can try to delete this file on the host and it will be deleted in the container as well.

Now run a new container, but specify a volume using -v:

docker run --rm -it -v /myVolume myTest

This adds a second volume and the whole system ends up having two unnamed volumes.
On the host-side, the new second volume is anonymous and resides together with the other volume in /var/lib/docker/volumes/.

It was stated earlier that the Dockerfile can not map to a host path which sort of pose a problem for us when trying to bring files in from the host to the container during runtime. A different -v syntax solves this problem.

Imagine I have a subfolder in my project directory ./src that I wish to sync to /src inside the container. This command does the trick:

docker run -it -v $(pwd)/src:/src myTest

Both sides of the : character expects an absolute path. Left side being an absolute path on the host machine, right side being an absolute path inside the container.

We run this command:

docker run -v $(pwd)/src:/src myTest

Is the specification purely informational?

No, it does mount a volume, it's similar to -v /var/www you just don't specify a mount point in the host machine, so docker will take care of it.

What are the (dis)advantages of (not) specifying volumes?
Why and in which cases should I use VOLUME and when shouldn't I?

It's probably a best practice to never use VOLUME.
The first reason we have already identified: We can not specify the host path.
The second reason is people might forget to use the --rm option when running the container, you will end up with a couple of unused volumes and it might be a daunting task to figure out which of all anonymous volumes are safe to remove

What about files that may optionally be mounted? Do they count as volumes in that case, too?

In this case you don't need volumes; mapping a single file does not make sense, I would use ADD or COPY inside the dockerfile.


How to use same Helm dependency in many charts?

Finnaly, I found the solution, thanks to @simbo1905.

All that I need to fix that problem:
values.yaml Child Chart 1 (or two)

postgresql:
  nameOverride: chart-1-postgres

In the Child Chart 2 nameOverride became to chart-2-postgres.


Best tool for Docker Container test automation?

Although we're deploying to Kubernetes, we've found that docker-compose is a nice way of running tests in CI. Either set the IS_TEST flag in your container or use a separate Dockerfile where tests scripts are wired up. That way you can just run docker-compose build to test, and pull in any dependencies you might need as well.


I've done this before, normally in the form of having a /.well-known/info endpoint available on every microservice that then returns a JSON payload with what's in it:
{
    "service_name": "token_exchange",
    "server_id": "RD1717628373",
    "commit_hash": "f72a28",
    "branch": "feature/content-security-policy",
    "version": "v2.8.12-f72a28-prerelease1",
    "dependencies": [
        {
            "id": "db",
            "type": "database",
            "endpoint": "db.mysite.com"
        },
        {
            "id": "service",
            "type": "microservice",
            "endpoint": "RD661872834"
        }
    ]
}

I then have a cron job and custom dashboard that queries all of the services that it knows about and then walks the dependency tree to add any newly added dependencies.
Considerations

Agree a Standard - The first step is to agree on a standard across services that you want to report on, then get them implemented.  It's hard to show the value of this kind of thing when only 1% of your estate implements it.
Start Small - I created a command-line interface that went and queried it at first.
Security - It could be an attack surface from an information security perspective, so test it and possibly add some authentication component to it and turn it off in production.


Is there an organization that offers a Hashicorp Vault SaaS solution?

It seems that Hashicorp itself offers an enterprise solution including 24x7 gold support called Hashicorp Vault premium enterprise.


You might not be able to get this done on the free tier. Puppet for example isn't going to want to start because of the RAM limitations. The AWS free tier uses thje t2.micro instance which only has 1GB of RAM. Your operating system alone probably needs 512 to run at idle. This leaves you a mere 512 MB or ram for all of the things that you listed.

While the issue with puppet not starting can be overcome by editing /etc/sysconfig/puppetserver and setting something like:

JAVA_ARGS="-Xms512m -Xmx512m"

(this issue arises because puppet expects 2 GB or RAM out of the box)

You are bound to find that running all of these on a single server is debilitating and painful with only 1 GB of RAM. You should probably consider using something like virtualbox on your PC or laptop so that you can get some more ram than that offered by the AWS free tier. I'd say you want at least 4 or maybe 8 GB of RAM. While this probably can be done, I don't recommend it. You may have to stop one service to run another which then makes it difficult to have one talk to the other (eg, ansible talk to Jenkins while puppet is running, for example.)


How to set a Git repository when creating a job in Jenkins from a Groovy script?

def repository = 'git@somerepo'
import hudson.tasks.Shell
job = Jenkins.instance.createProject(FreeStyleProject, 'TestJob')
job.setDescription("Some description")
job.displayName = 'SomeTestJob(TESTING groovy)'
job.scm = new hudson.plugins.git.GitSCM(repository)
job.scm.branches = [new BranchSpec('*/master')]
job.save()

Thanks a lot Michael Durrant for giving that link !


How to use in Chef different attributes values for different nodes

You may use attributes precedence for that or a wrapper cookbook, I highly encourage you to take some time to check out https://learn.chef.io which cover those uses cases.

This is a kind of antipattern, one shot configuration are not what Chef is for, but for this use case I'd go with a common cookbook doing what is needed and a wrapper cookbook defining the attributes and including the common recipe.

A typical wrapper cookbook would need a few files:

metadata.rb as another cookbook with a line depends 'common_cookbook'
attributes\default.rb with the values for this type of server, same syntax as in common_cookbook
recipes\default.rb with the line include_recipe 'common_cookbook'

This pattern is also known as role cookbooks, more details in this blog post


Jenkins pipeline: do "Declarative: Checkout SCM" step only

As implied by the links in comments, for better or for worse, this behavior is an unavoidable consequence of how Pipeline is designed.  Pipeline configuration changes in your Jenkinsfile cannot be picked up by Jenkins without executing your Pipeline.  This includes things such as adding parameters.

My personal advice is to ensure that your Pipeline jobs are non-destructive, or in other words, to ensure that the worst thing that could possibly happen by running a job is nothing.  This allows you to pick up configuration changes by just running your job after committing the changes to your Jenkinsfile, and then you can simply abort the job (or let it run to completion if it's a short job) with the confidence that you won't break anything.  Then subsequent runs will have your configuration changes available.


Should Jenkins apply round robin if nodes have the same label?

@jayhendren indicated that:

  I think you might find the answer to your question in this SO post:
  https://stackoverflow.com/q/11024007/2777965